You could just say that you have already made plans based on the pre-arranged custody schedule and that you´re sure he´ll understand. My worry would be that the feeling of being a chump for helping them have their romantic get-away would eat away at you, so as to preserve your equilibrium you won´t trade so that you can maintain your sanity- as much as one can in the circumstances. It may look like to them like you being difficult but they are not thinking at all about you and your situation, plans or feelings. You´d actually be doing him a favor to decline as he would learn to plan ahead.