Re: red flags everywhere or am I crazy?
I am a dissenter on this one. Your wife's story is plausible. It also seems she has weak boundaries to some extent by not totally blocking him on her phone. If the guy had a history of harassing women, he could be the kind who grooms women over time. His actions sound like such a person.
Would you describe your wife as someone who tends to try to please others? Does she tend to do things for others when she would prefer not to?
Her ex-boss certainly seems like a persistent creep. I am not leaning towards your wife having had an affair with him.