You have gotten some really good advice from others dealing with the same issue but I wanted to let you know that I think your idea of writing down some of the mean or hurtful reactions may help you when you talk to him. It is possible that he doesn't realize how those sound.
I also really agree with the fact that he is probably really not happy about how his low drive makes him feel. Maybe he reacts the way he does because each time you initiate it reminds him of his low drive. He is directing this anger towards you and it is not fair.
I hope you get this worked out - you are taking great steps in taking control of the things you can and working on making yourself happy.
I will never say no in 2012 