There is no comfortable way to say this so here goes:
The fact that this would be an interracial relationship puts a whole new light on the situation. Does he date sisters - does he know if you date white boys or not. This may be his hesitation. That, with the preception of you pulling back may have made him concerned that you are not sure of starting something more serious than a friendship.
just a thought
