Milly, I agree with everyone else about the graduation -- please go! I know that is direct, but hard as it might be to think that they might be talking about you, I would venture the view that it would be better that they talk about you holding your head high, being polite, and honouring your D's achievement than thinking that they can intimidate you into not coming.
It doesn't mean you have to pretend that everything is OK, not at all. But that day is about your D, if they want to talk behind your back let them, it is they who are behaving badly in that case, not you. Hard as it is, don't let yourself get drawn in....
I, too, have had my H say to the children that he wouldn't come to events if I were there, that seems to wax and wane with whatever else is going on in his life. The trusty "I'm sorry you feel that way" comes in handy here, or better yet, just silence. That works wonders, and I mean for us.
This is hard, very. But standing up to bullies pays dividends, and sooner than we might think. One thing history has taught us in all respects:
Appeasement never works.
I also agree with not speaking to him directly about anything legal btw, that tends to backfire on us....