Oh, yes,Elle, I agree with BLR, now you have said this.
I'm not suggesting that he is racist, because he obviously likes you and, as you say, he mixes with various cultures at work.
But think about this:
'I haven't ever invited to him anything besides my brother's b-day party which he said no to because he knew he'd be the only white guy there'
If he became your boyfriend, he might have to attend various family functions with you, where 'he knew he'd be the only white guy there'.
If this bothers him then this may be what is worrying him.
He may even worry that dating could lead to something more serious and be worried about the consequences ~ eg mixed-race offspring.
Maybe you need to have a quiet serious chat with him to sort things out about how he feels and how you can deal with it all.
