Re: How concerned should I be
It may be nothing. But the two things that bother me is her getting so defensive about it, and not telling you when she went to see him. For this type of thing to work, she has to be pretty transparent.
You may have to accept the fact that there is another man permanently in your life due to the children. This is a very gray area here. You'll have to learn to trust her, and just as importantly, not show jealousy when it starts to eat at you. Look for signs though, and don't get blindsided. When in doubt and you have justifiable suspicions, confirm if they are true or not. Then put it behind you.
IF nothing is going on, then I see nothing wrong with bringing the entire family together for special occassions or just a bbq or whatnot. Sometimes you have to put aside personal feelings for the kids.
This takes you being very secure in yourself and the relationship though. And it also takes understanding from her that this is a bit of a hard situation for you, and she should be a little understanding when you occassionally have questions or little doubts.
I encouraged my W to maintain a relationship with her ex because of the kids, and also because the bitterness and battles were eating away at her happiness (and in return ours to some degree). And it was hard for her at first, because he was very bitter. She had to take a lot of crap from him and "turn the other cheek" before things settled down and he got tired of fighting someone who wouldn't fight back. Now all is seemingly good. They visit and do things as a family regularly. She's anxious for the day when enough time has passed that we can all visit together. And I'm confident I have not a worry in the world about any of this. Sure, I still get a nagging twinge of "what if" from time to time, but she's never done a single thing to make me doubt her. And believe me, I'm very in tune to possible "red flags".
They talk, they text, they visit. She'll always take his calls in front of me when I'm around. She keeps me in the loop. She knows to lie to me about anything involving another man would end our relationship, so she does not do it. I resist questions as much as possible, and don't pry too much, and never act jealous. If I have a question, I ask it, and I get an answer.
Could this backfire on me someday? I suppose. But I don't think that would change because of WHO the other guy is, but rather who she is. She will be pre-disposed to that type of thing or she won't be. I have zero cause to think she is that way, and I certainly won't push her towards it by being insecure, controlling or jealous. By being this way I feel I've done much more to maintain her loyalty than by not trusting for no substantive reason.
If there are red flags, find out what is going on. If you find out it's nothing, you will HAVE to put it behind you. There is another man in your relationship, and because of the kids, he'll be there a while. The wrong thing to do is to try and create strife between them or tell her you won't accept this extended family type relationship (assuming he's not some evil child molestor or abuser). Being controlling will only speed up (or perhaps facilitate) an outcome that otherwise would not happen or would have happened no matter what you did.
Trust until she's proven she's unworthy. And if that's the case, "next" her sorry butt and move on.