I think asking for "not a drug user" is not being too picky. If that's something you feel strongly about, you have every right to live a drug free life. It would be the same thing if he smoked cigarettes and you were against smoking.
That being said, many college students try out weed and then give it up, it's a passing thing. So you might want to keep him as a friend and see if he does that.
