emuna - I'm offering my opinion a little late here, but I also doubt your husband's motives.
When both partners in a relationship are in a good place trust-wise, they are able to be truly honest with their partners. They can be honest (without worrying about shame or judgement) about both their own fantasies and their responses to their partner's fantasies. Depending on how that conversation goes, it becomes clear whether any further action/decision could be taken.
But, from what you have said previously, your marriage is not in a place like this at all. You have clear (justifiable) trust issues that you have been trying to resolve for several months now. In your current marriage climate, I'd interpret your husband's sharing of his fantasy as not just insensitive, but offensively disrespectful.
FWIW, I also think that he is trying to find a way to have his cake and ... actually have another cake as well. Have you confronted him yet with what you know about the actual substance of his conversations with the OW?
If I had fantasies involving my wife and/or I, I'd be able to tell them to her because we have a mutual trust. In OP's case, her husband does not have her trust. Worse, he knows this (or is too dumb to realize it). Therefore bringing it up now is only self-serving. Even if OP was into this, the timing isn't right.
He's brought it up now for one of two reasons. Either what others have said above (it's, in his mind, a way of equalling out his lust for another woman and/or a bridge to "allow" him to sleep with somebody else) or the marriage is on the rocks so why the hell not?
About 2 or so years before my XW left me for another man, she told me I was 'allowed' to get BJ's from other women, if I so chose. At that point in our relationship, she was not able to give me oral sex because she had TMJ/lockjaw (she really did). This was her justification for 'allowing' me to get this elsewhere - she said. I obviously declined, but simply did not see through her real motives.
At the time, I genuinely thought she was just being nice, and they were just words, and obviously I would decline (had no interest in that, anyway). In retrospect, it's clearly obvious it was her trying to balance something out. Whether she just lusted after somebody else, or she had been physical with someone, who knows. I should have known better, though.