I'm not even familiar with the word, (And whether I'm 'white' is questionable, depending on who you ask, but I look caucasian enough to be classified as such in the States, I guess) but I wouldn't like it if I was just referred to by skin-colour.
But it depends on the context.
If the word was used in a group where everyone else was black, for example, it would carry the meaning of 'you're not one of the group', which it wouldn't if it was used in a context that was more diverse. (And that's also why slurs against minorities are more problematic than those describing majorities; they are used to define someone solely by their skin-colour/ethnicity/sexual orientation)
Also if a politician used it in a speech or something, I'd probably be more annoyed than if it was a random person on the street.
Abomination said:
And even if not I really wouldn't care... I actually don't mind slang terms for races. We call Americans "Yanks" and the English "Poms" and they don't take much offense.
Since when are Americans and English races?