It's not the vegetarians themselves that people hate.
It's when the same vegetarians start telling you that you are murdering animals, abusing animals, and start generally pushing vegetarianism on you.
But, hey, the same goes for any person or group with views different than the majority.