It depends on the game. I loathe shooters but if there are elements of the game I like, such as in Mass Effect, I'll keep it on the default setting. (I'm much too proud to ever play on Easy even if I'm not playing for the gameplay per se.) If it's a 2D platformer like Mega Man then I'll take whatever ridiculous challenges the developers can throw at me. I'm also the type to play through an RPG underleveled so that I have to rely on my wits to carry me through battles I'd otherwise steamroll.
Another variable is the kind of challenge that a game offers. Take God of War-it doesn't really get more difficult, it just gets cheaper. The enemies take longer to kill, they take more of your health with each shot, and it's far more likely that you'll get stuck in a kill cycle (ie one move will chain into another which chains into a killing blow and there's nothing you can do to stop it). Your only recourse is to block a lot. You don't need to become more skilled, you just need to play very defensively and keep your combos to a minimum so that you don't get stomped while you're doing your best impression of a blender.
Worst of all is how minor quibbles with the game mechanics that you can completely overlook in normal mode will screw you over on the harder difficulties. I tried playing GoW 3 on God Mode and I got stuck in the maze when I kept pulling out my bow, intent on shooting the enemies across from me, but Kratos turned around to target an enemy that was behind him and I couldn't efficiently switch to the enemies I wanted to kill. I could have gotten past it but it just wasn't worth it.
Give me better AI, test my memory, test my reflexes, test my resourcefulness, but don't ask me to work with wonky mechanics or meathead AI attached to a slightly more durable puppet.