First off, did you play Morrowind? If so, this is going to make a lot more sense. I don't have a lot of time right now, so I'll give you the short version.
If you're interested in the whole story, go here: http://www.uesp.net/wiki/Skyrim:Battle_of_Red_Mountain
The Dunmer, as we know them today, actually started out as golden skinned elves (kind of like the Altmer, but less yellow more gold). However when the Tribunal, who were the council to Lord Nerevar, used Kagrenac's tools to attempt harness the power of Lorkhan's Heart, "then, in that moment, all Chimer were changed into Dunmer, and our skins turned ashen and our eyes into fire".
Why this happened is up for debate. The Tribunal, by using the tools, were betraying an oath they made to Azura. When she confronted them, they scorned her and proclaimed themselves the new gods. It was at that moment the change occurred. This has led some to speculate that it was a Curse by Azura.
However, she replied "This is not my act, but your act. You have chosen your fate, and the fate of your people, and all the Dunmer shall share your fate". Suggesting that it was the use of the tools that caused the change. However, it should be noted that she never outright denies cursing them.
That's the short version. I suggest checking more sources for the full version. Like I said, it's very interesting.