Why are you allowing your fiance to usurp your parental authority when he's only been a member of your household for 4 months? You said you went to pick up the kids early "to appease the situation." This isn't good at all. Your kids are going to realize that you're handing over your control to this new guy (who, frankly, is behaving like a jerk to them at least part of the time). That's not going to make them feel secure.
They have enough to adjust to with the divorce, accepting a new person into their family, building a relationship with him, etc. right now. He needs to focus on building a foundational relationship with them at this early stage, not on being the disciplinarian. That's still primarily your job. There has to be a chain of command when it comes to parenting in a blended family. That chain of command goes: You and Dad > Stepparents > Kids. All authority he has comes through you. Even if he takes away a toy the kids are fighting over when you're not there, it should be because you guys have discussed what discipline you as Mom are comfortable with
ahead of time and agreed on what he should do in certain situations.
I'm guessing by your description of his behavior that he has no children, himself, or at least not any he's had much experience living with. Your children might not notice the nuances now, but they will, and sooner rather than later. Kids are very sensitive to tones of voice and tension in the household. Pretty soon, they'll feel like they're walking on eggshells in their own home, too, and that will lead to all sorts of resentment that can ruin your family.
I'd nip it in the bud now. Sit down with your fiance and let him know that you want him to focus on building a relationship (not a Dad-type relationship. Just a relationship.) with the kids right now. Offer suggestions on how he can do that. Come up with an agreement on what your expectations for the kids are in the house, but beware changing them too much, too quickly...better to work on one small thing that annoys him at a time. Simple expectations are best. Ask him to play a supportive role and leave discipline to you when you're there. Ask him to agree not to argue about parenting issues in front of the kids. If a new problem comes up, he should discuss it with you privately and let you decide how to handle it. Of course, he can give suggestions, and you as his partner should consider his feelings on the matter. If you're not present and he's watching the kids alone, discuss and agree on how he's allowed to enforce the expectations.
There need to be clear boundaries in place. For the kids AND for your fiance. He will also have his own boundaries to enforce, as well, but they're going to have to be realistic ones if these are young children. He'll need to learn how to pick his battles. Hopefully, he'll grow to love your kids and vice versa, and in a few years, his relationship with them will be established enough for this to all flow more easily and organically. It's just finding the right way to mesh together all of these boundaries in a way that's fair to everybody.