Re: He says he wants divorce, I don't think he does and I sure don't
Considering he has lived in a seperate state for 6 months, I doubt he'd get full custody unless you screw up. One possible way to screw up is to give the impression you are denying parenting time. You don't want to be accused of parental alienation. BUT, you can certainly tell your STBXH that you cannot take a trip to his state due to work and finances, invite him to visit the child in your state, and offer to have your child use Skype to talk to him.
I have the feeling his text about visiting with your shared child was more or less him trying to get evidence that you are inhibiting parenting time. Be very careful how you respond to anything and everything sent by him or his family/friends. Anything you say can and will be used against you in court. Anything they can do to make you look unstable or like you hate them and will poison the child against them may come back to bite you in the azz, even if it wasn't your intention. They can take things out of context, so be very careful with phrasing.
You are now at war. His family and friends are the enemy. Delete them from social media. All of them. Ignore their emails and texts. DO NOT get sucked into arguments, debates, or drama.
By the way, I'd bet my last $5 your STBXH waited until the near the 6 month residency mark to file on purpose. I'd also bet his OW is his daycare plan.