Honey,
You need to do whats good for you and your daughter, and that is letting this man go. Eventually the fights will get worse and maybe even abusive if they haven't gotten there already, soon your daughter will be brought into the situation, again, if she hadn't been already and that can REALLY damage her later in life.
Thats my 2 cents, I hope it helps
