Re: Don't know if I should...
It sounds like there are a number of issues that need to be considered.
Your husband is an anxious man - I personally have seen how debilitating anxiety can be in general but especially in combination with other problems. In his mind he probably is trying to provide for a family while remaining in control of everything and not being exposed for any weaknesses ("stigma of taking medications").
Add to this erectile dysfunction and you have got a recipe for disaster - he probably sees sex as his biggest weakness exposure hence the appearance of not caring and no sex for 5 years. Deep down I would bet that he cares very much - just doesn't show it.
He (in your words) is otherwise a very good father and person. He is crippled by his anxiety shame and desperately needs help but will not get it for those very reasons. That is your challenge.
Now you say the affair isn't the issue here - I beg to differ. It is a very important issue here because it defines you! While your husband is anxious and withdrawn, you on the other hand are a liar and a cheat! Probably one of the worst types of people your husband should be with. You certainly have done a good job of justifying your affair and blaming everything on your husband. I understand that you are suffering from loneliness and no sex but then you should have divorced him. You didn't because of $$$ etc not because of the "in sickness and in health" part of your vows. So both of you need serious help. Him with his anxiety and you with your very serious character flaws.
I would divorce him and be fair in the $$$ split so that you can go on to be happy and so can he! The threat of divorce may force him into action to get help but even then, I suggest going through with the divorce. You two are very mismatched. You married him for stability even though you were concerned early on that he couldn't keep it hard.
This is my quest, to follow that star
No matter how hopeless, no matter how far
To fight for the right, without question or pause
To be willing to march into Hell, for a Heavenly cause