Doormat route for 4 years failed miserably for me. Just gave him time to plan his exit his way.
It's your life, your marriage. You have to do it the way you see fit. But A LOT of us began as understanding, forgiving, all those great traits you are showing.
Maybe get a copy of the vows you actually spoke or agreed to from your clergyman, and take that with you to counseling. Then the 3 of you can discuss what they mean, and if BOTH of you are still committed to them all. Just an idea. Good luck.