| | Re: Will he ever forget?
Tell him that if he brings up "the boyfriend" again, you're out of there.
It may have been an EA (from what little you've describe it) but there's nothing wrong with being a friend that someone can lean on a bit during troubled times.
From your description, that's what it was and your H bringing it up again and again is him acting like a little kid. As for him forgetting/remembering - it depends on how HE saw the relationship. Maybe that's something you two need to talk about. Calmly!
Why are there trust issues? Because you supported your friend when he was down? Personally, I'd be glad that my wife/GF/SO was that much of a friend to someone.
It ain't illegal until you're caught!