This guy sounds like a complete creep.
Agreeing with everyone else - I don't know how you couldn't afford a divorce - but whatever. The sooner you aren't with him the better.
Finally, I wish that more low drive spouses posted on this board. It would really add some light and life to the same thing about not getting 'enough' or 'any' and us all comforting and commiserating with each other.