Empire. Biggest deciding factor for me was Ulfric's use of the Thu'um to pop the (former) high king... from what I've seen/heard in-game, Ulfric challenged said high king to a little duel, he accepted, then a single Shout ends it... messily.
Greybeards say the Voice is to only be used to praise the gods with only Dragonborns being exempt from that. Was Ulfric a dragonborn? Don't think so.
On a more personal level, had my Khajiit thrown her lot in with the Stormcloaks, she'd basically be fighting for people that would ultimately shun her and any others that aren't Nord and that share their same ideals.
I guess what it boils down to for me is you're either fighting for the Nord's freedom with the Stormcloaks, or fighting for the freedom of the other nine races to live and work in Skyrim without being ostracized and ridiculed.
In a way, the Stormcloaks are a lot like the Forsworn; natives fighting for their homeland. Which is a noble and just sentiment, don't get me wrong, but the Forsworn are just... wrong as a whole. Nords aren't quite as bad, but their motivations just seem culturally selfish. /:|