Speaking from a guys view, If you accept his invite he will think he can get back with you or persue a sexual relationship with. I know that sounds silly and shallow but this how a guys mind works if i am being honest.

If you want to persue something with him then accept the invite. If not and you want to move on, ignore the invite and block him from your list.

All the best.