Communication is the Key
It's difficult when you have children and are feeling lost in your marriage - especially when you feel like there is a problem but the other half insists there isn't.
You're not crazy for feeling like there is a problem - when he calls you names that is not showing you respect, and guess what? That behavior will translate right on down to your children. They repeat exactly what they see, because they see it as accepted behavior. Not only does that affect how your child treats you in everyday interactions, it will affect how your child interacts with their peers, and in the future their own spouses. Yes, I speak from experience.
My husband's father was abusive to his mother (they are divorced now), so how did my husband treat his mother? With the same disrespect! He still doesn't respect her. He says he respects me, but he will still have behavior towards me that is disrespectful at times and I worry about when we have children how he will regard me in front of them.
When he says he hates you, whether or not he really means it, he's saying something that will hurt you, he is being malicious, and he is being unfair to you. No one in a friendship, let alone in a marriage, deserves this kind of language. He may later say, "I'm sorry," but unless he ceases that kind of language, he obviously doesn't understand the impact that his words can have.
Additionally, a husband should know when it would be harmful for his family to take a trip with friends - in this case he is only taking himself into consideration and he needs to understand that not only is this inconsiderate to you, it's inconsiderate to his children and as the provider for the household he is abandoning his responsibility to his family. Sure, he makes the money so he should be allowed to have fun, but obviously a trip to Las Vegas is not fun within reason. Going bowling or going out to eat with friends is fun within reason as far as your budget is concerned. It is irresponsible, it is selfish, and it is poor parenting on his part to even consider such a financial burden as his right.
You're wondering whether or not it will be harmful for your children if you leave. It is harmful to be in an environment filled with anger and tension. It is harmful to see Mommy sad and to not know why or what to do. It is harmful to hear Daddy yelling hurtful and mean things. They are young now, and this is the opportune time to start anew - once they are older and in love with the idea of family, that is an extremely difficult time to tear them away from your husband/their father. I should know. My parents separated when I was in eighth grade. I'm 23 now, and I can tell you it changed my life forever in a negative way.
As a child, even when the child doesn't blame his or herself for the dissolution of the marriage, there is a feeling of loss and a feeling of no control. There is nothing they can do to reclaim that family, there is nothing they can do to make their parents love each other again, and there seems to be no reason for all this hurt. All the child knows is that he/she is hurting, and whether or not it gets better it will never be the same.
An extremely young child (1 and 3, like yours) will adapt more easily to change and will benefit from being removed from a negative situation.
If you want to wait it out and see if it will get better, it won't. Remember that story I told you about my husband's parents? They were married for over twenty years - but they also had the same problems for over twenty years. Not unless you guys have a breakthrough in how you communicate will things change. Whether that be through counseling or a heart to heart talk or shocking him into realizing that he is losing his family is up to you and him. It is up to you to open that line of communication (since obviously he doesn't see the need) and it is up to him to respond appropriately. If he values you and he values his family, he should see that something needs to change. You don't need to change him and he doesn't need to change you - you both just need to learn how to better communicate and how to show each other love and respect. Ultimately, that is what marriage is all about.