Oh gosh...it can be so hard to see what's really happening when you're IN the situation yourself.
It was a smart thought to come here to inquire. We are not involved so we have the ability to assess differently, more objectively.
These are not things you hope to hear, but you need to hear it.
The threat you're feeling is legit- absolutely!
See, sex appears to not have a deeply emotional draw to you, or so it seems. That is how you are able to share that part of yourself and your husband without worries.
However, time spent together, talking, relating, learning about one-another, etc...those are bonding activities. I am so sorry but this is bad. Seriously.
Please, if you value your marriage and relationship with your husband MORE than you value casual sexual relationships and swinging, then do yourself and marriage a MASSIVE favor and stop the contact between them.
What is happening between them is how two people fall in love. It's happening. You see it and don't fully want to believe it, so you're in mental turmoil. Face the fact.
Be aware of the reality and END the contact now if you want mental clarity and peace...and HIM! As it is, it will be hard for him- sounds like he is alread starting to fall- GO CATCH HIM!!!