It's not so much that they are necessarily dicks, it's more the fact that they merely wanted a friends-only game. It's happened to me before--the reverse; I've been playing a game that was specified as "friends-only" in the lobby, but because it was hosted on a non-official server, it made no difference, and others joined anyway. That's the downside of playing on a non-official L4D server (at least on PC; I've never played online 360 L4D).
So if any of you have jumped into my games, and immediately received a vote-kick, it's not because I hate you, it's because the 3rd-party server overrode my "friends-only" option.