Gigo, I'm so so sorry for you. For what it's worth, I'm new here, so take everything I say with a grain of salt.
PHTlump sounds possibly correct - my husband also withdrew emotionally and blamed me and made me feel for years that it was me, that I was unattractive, etc, and I never suspected infidelity but now I see he was probably having one-night-stands for years. I just never saw it.
I also gained weight during pregnancies but lost most of it after and it didn't matter. I've finally realized that it wasn't about me.
I'm sure now (and suspected for a long time) that my husband projected his mother onto me - meaning that he looked at me and "saw" his mom - probably ever since we got married. (I had my first baby 9 months + 1 day after our wedding, so it was easy to get confused about what was the cause of the trouble
From the time I believe (now) that he started sleeping around he got exactly like you describe your husband - cold and distant and demeaning. Little or no sex.
Tell me: why would a man with a "good heart" put down his wife and the mother of his child?
A very smart lawyer woman told me recently to "shake it up a bit." Don't be so available to him. Why are you cooking for him and hugging and kissing him and trying to make him feel important?
Let him come home one night and find that you're out. Take your baby and go to the mall. Go out with friends and get a life.
Good luck to you and I hope you find happiness!