Black people didn't oppressed or enslave other blacks in America like whites did. There wasn't black supremacy, blacks were relegated to second class, not permitted to go to schools with, hospitals with, travel and eat with, or otherwise mix and marry with white people. It was against the law to marry blacks in America less than 40 years ago.
I think it's pretty boneheaded for anyone to use that kind of language to address others, I see whites calling other whites that word, blacks doing it, whatever the case may be. Fact is though, white people used it disparagingly (and unfortunately still do all too often) against blacks more so than without pejorative intent. As a general rule, as long as racism continues to exist as it does, we will still be offended when a white person calls a black person by that word. It would be unreasonable not to be offended when someone n-words me, and white people have right to my face without batting an eye.
I've been called that by blacks and whites, trust me when I say the difference is quite palpable.