Jon offered to stand outside splitting wood and looking angered

Tonight was the first horrible interaction my husband and I have had.
It was his night to pick up Esme, she will spend two and a half days with him, and ever since he suggested this new schedule I have not felt good about an evening pickup. In the evening there's a routine, everything is winding down. It is not when we start new things the morning is when she is prepared for where she will spend the day. In fact we begin every morning with a conversation that involves "what kind of day is it today mama?" To throw that off and switch gears midstream with a night time transition is disturbing. For all of us.

Tonight Esme decided, via breakdown in the car, that she was not going to have a daddy night. I informed her that for that to be so she would have to call her daddy and discuss it with him. At that point she said "just forget about him mama."
I turned down the radio and countered with, "well, I don't think you want to forget about him, that would be very sad. I think you need to decide. If you want to forget about calling him then you should just be happy when he picks you up for your time with him. If you want to forget about your daddy night then you will have to call him. But you cannot 'just forget about him".
She decided to call him. Took the phone into a separate room from me and apparently told him she didn't want him to pick her up. Whatever the exact conversation, it ended up with her bringing me the phone as she collapsed in tears, and him telling me he was coming to pick her up.
That interaction was followed by a text that said "On my way. I expect you both outside. Or i'll come yell at you inside."
I attempted to steel myself and headed outside to meet him. Our conversation was not pretty.
He started it off by telling me how patient he has been and how that is over. "this thing we are doing is 50/50."
My response was that his "walking out on our family and not even having the decency to tell me he was thinking about walking out until months after he had actually gone, was not at all 50/50. That was 100%.
It was his decision to leave the family and my decision to make sure he gets as much time as he does with Esme."
His response was "I have been patient with you because you're hurt or whatever (that was my favorite line in the whole conversation p.s.) but I am done being patient."
I inquired as to what point in our entire relationship he had ever been patient, much less in the time we've been separated. His answer: "The fact that you even think I'm going to bring Esme back on Saturday shows exactly how patient I've been."
I am proud to say this is the first conversation where I have actually to his face called him an asshole, as in:
"why do you even have to be such an asshole about this." Then I said, that when he shuts down his daughter when she tries to voice her concerns, and threatens over text message to yell at her it does not bode well for his relationship with her in the future.
There was then a bit of door slamming as I went inside to retrieve the child. My friend (God bless her and her eternal patience with this situation) had to walk Esme out to the car so that I didn't make her feel worse about having to go. Seriously the last thing I wanted to do but I was understandably having trouble composing myself.

Then I was left to cry and ask myself how my life got so fucked up and why I ever thought it was a good idea to marry him in the first place. Yes!! I am well aware you have all heard this enough but trust me you have heard a minor fraction of the times I've berated myself for it, so thanks for hanging in. Seriously Thanks!

The part that saddens me the most is that I was feeling pretty good about her leaving tonight. I was not feeling nearly as distraught as last week. I thought it was going to be okay.
The only reason I even let her call him is because she felt so strongly and I'm tired of being the only one to talk her into spending time with her dad. I wanted him to hear how she feels and patiently convince her of how much fun they would have and how much he loves her and wants her with him.
His reaction just convinces me even more that he is using her as a pawn and that's not what I want to see or think.

Please feel free to share your insight. You can judge, advise, berate. My friend already told me I should never have let Esme call...you can't tell me anything I don't fear and contemplate myself. What would you do in my shoes? How am I doing? What am I missing? What will I see a year, two, five from now that I will wish I knew now?

AAUURRGGGGHHHHH!