Sometimes it's not about not wanting to be a father, but about losing what was in your previous lives.

You can say it's selfish, but it is still a feeling, but he might resent the fact that he now needs to share you, or is now not the most important thing in your life. Usually these feelings will go over time, as he becomes more secure in the new relationships.