I hear you loud and clear. This is all part of the process. You are distancing yourself because you are trying to protect yourself. You don't want to be hurt again and you fear that you will be when you have an argument with him and he says things in the heat of the moment.
I've been there many times and still get there - just a couple of days ago actually.
Whether you work on it some more or decide to leave it up to you, but know that this is a process you need to go through to heal. My advice is to wait it out and see what your feelings/emotions do with this situation. They may go away or they may not. If they don't go away, then you have your answer. But if you go back and forth with it, you'll know that there is still hope.
This is a traumatic experience we've been thrown into by the person we love immensely. It's no wonder that we have turned insane because of it. I hate the person I've become but we are truly shaped by our experiences... hopefully, future experiences steer us back to the way we once were....
I have a blog that discusses my journey through recovery, when you have a chance check it out, it may help to hear about mine and others experiences...
Infidelity Rage
Hugs...