Hating Black people because they are black is crazier, everyone knows that black is cool, check out Fonzie's Jacket or look up Henry Ford's offerings of colors for the model T.
Yellow on the other hand is the color of madness, and must be hated by all those who wish to be free of the corrupting power of the King in Yellow and his master's infernal Yellow Sign.
EDIT: also this Yellow [http://psychology.about.com/od/sensationandperception/a/color_yellow.htm] can cause feelings of frustration and anger and is the most taxing color on the eye, so again yellow is a terrible color and thus hate for it is most rational.