I think that it's the mistreatment of a specific group of people that is the problem. I mean, racism has been a part of hate crimes, assults, and even genocides on a scale that misanthropy has never matched. If you are a racist then you are part of a group who feels the same way, a group which can get large enough to cause some serious damage. Misanthropists can cause harm by themselves, yes, but they aren't going to ban together to cause extreme harm. On the scale of worries, racism is much higher.