Either you will go to the ends of the universe for your baby, or you won't.
As I see it, you can stop supporting four people, yourself included, or you can get your own life together. Your wife will be welcome to come along, and that's all. It's down to your wife deciding and you leaving. Good luck,