I have such a clause in my contract too. I think this is only becoming a problem at all when they want to get rid of you. As long as they want you, it would be very stupid to enforce this since your productivity could go way down when they tell you what to do in your free time.
I've always refused everything but the non-compete and "do not use company resources" clauses on the principal anything else is just rude and controlling (and missed out on a few jobs because of it).
Maybe we should, as a collective, start a campaign to drive out the most over bearing clauses!
