Unfortunately it is beyond unethical for a doctor to ever have a relationship with their patient. I'm not sure if you're in the US, but AMA guidelines are extremely strict about that. There are many reasons for that, and part of it falls along the same lines of why a teacher cannot date a student and a boss should never date a subordinate. There is a "power relationship" there. It influences the relationship between the two people.
Especially with doctors and nurses, it is a well known and well studied phenomenon that the patient often falls for the "caregiver". The caregiver is focused on the patient. The caregiver is listening to the patient. The caregiver is helping the patient feel better. All of these create a romantic atmosphere. That is why it is absolutely forbidden.
If your doctor is allowing this to escalate, that is just as bad as a high school teacher encouraging or prodding along a student who has a crush on him. I'm not saying that you're a kid - but that he as a doctor has been trained in this very situation and KNOWS it's unethical. For him to lure you along is just wrong, wrong, wrong.
I would steer far clear of him. If he's willing to cross these kinds of ethical boundaries, that's a giant warning sign.