I'll drop right in here to say the in their private life argument doesn't fly here. Because the complaints aren't about his private relationships, they're about public and professional behavior.

Frankly the callousness of people like Asimov bothers me. You're left with two possibilities. Either they simply don't care that they are making the other person uncomfortable, angry or scared. Or worse that they're getting off on that.