You both need marriage counseling. Then he'll have someone to talk to.
Sounds like there's a lot of resentment on both your sides. The fact you mention that your relationship started with a lot of "on/off again" and contention is not a good indicator for a healthy long-term relationship.
Your concerns are valid, however they are coming from a place where you feel that you have made a sacrifice by giving up your male friends and becoming somewhat of a hermit. You should never feel resentful of choices you make for the sake of a relationship. There are differing opinions on this board as to whether you can have same sex friends. Personally, I feel that men and women are incapable of being friends. I used to have many guy friends during my first marriage and it created a huge issue that I was unaware of (and rather blind to). Now, I have no male friends other than the occasional male acquaintance I happen to work with on client engagements. The work is short term and I don't associate with them outside of the engagement.
Maybe he never quite developed the full trust in you. That could either be because he knew you were hesitant to give up your male friends (i.e. resentful) OR he is inordinately jealous and needs to work on forgiving you and moving on.
Again, marriage counseling (MC) will help to address all of this.
I agree about men and women cannot be friends. I view women as potential mates as I am genetically programmed to do. I can like them as people but will never hang out with women like I do with male friends. As far as trust goes, I think that a healthy dose of mistrust is a good thing. The two times I fully trusted a fiancee and girlfriend, they cheated because I was too trusting and was OK with them having male friends at work or hanging out with girlfriends who had boyfriends, etc.. This is the main reason why I do not believe in monogamy. I have been in an ethical non monogamous marriage for most of the 44 years we are married.
I have seen a lot in my life. Wives of friends offering me oral sex or touching me inappropriately. Married women who have been cheating on their husband for most of their marriage. A friend who was cheating from the time his wife was just a girlfriend through engagement and finally marriage. Every boss that I had, male of female, was cheating on their spouses including my current one which I found out about and now have job security as a result.
I travelled a lot of business and a lot of other business men I met at hotel bars, conferences, etc., found this to be an easy and safe way to cheat. My best friend let his wife hang with her group of male and female friends and guess what happened, she had an affair with one of the males and divorced him. When the guy dumped her, she went back to her husband who, lets say to be kind, was not the type of guy that attracts women.
We all get attracted to others. The only thing that prevents us from rutting in the street like dogs is our moral code. Most think their code is the correct moral code and judge others by it. You can still be a good person, but have a different moral code. Given free reign, a spouse can more easily cheat. Moral code or not, when emotions come into play humans tend to make made decisions. A watchful eye can re-enforce your spouses moral code. I was too trusting, and that never worked out for me, sexually or professionally. I am not talking about being suspicious of every little thing, but a healthy dose of mistrust that does not cause problems, and does not allow you to believe 100% can keep a trustful mate, trustful.