Except, you're wrong. When someone add the qualifier "All" at the beginning, they are, in fact, referring to all members of that particular group.
If they wished to NOT implicate all members of a group, then they should not use the world 'all'.
Besides, I honestly can't understand this rational of "Well, his insults don't describe you, right? So why are you offended?"
That's absurd. If someone said, "All black people are rapists!", you wouldn't say to someone who was offended, "Are you a rapist? No? Then why are you offended? They clearly don't literally mean all black people."
No, you'd berate the hell out of the jerkoff who said the insult. You wouldn't sit there and rationalize the comment until it seemed benign.
Personally, I don't care if you, or others, are fans of Bob's work. I hope he continues doing what he does and that people can enjoy it as they please. However, please stop trying to rationalize or excuse his abhorrent behavior. It serves no one, least of all him, and he'll never learn to 'tone it down a bit' unless he understands the harm it causes. To him, his fans, and those associated with him.