I play most my games on normal mode, might even bump it up to high if I really feel like a challenge (finished all the Gears games on the second to hardest mode so far) however there are exceptions here and there. Oblivion seems to get a common mention here and I confess I also bumped it down to the lower end... However it wasn't because the game was difficult... it was mainly because the fights on the normal mode were long and tedious. I didn't find a challenge in them, I just found them boring... if I can't one shot even a rat then we're going to have problems.
So yeah I tuned down the difficulty and just played for the enjoyment and believe me, I played that game... I played the shit out of it. Though, you know... I'd have made a compromise with the game, I'd have bumped it up to the highest difficulty if it meant that both I and the enemy had relatively low health bars. That would have been a fun challenge and would have relied more on skill when playing. However no, it just means you die quicker, they have rather sizable health bars (yes I know there are no bars on this game but you get what I'm saying).
I honestly find a decent difficultly is when they turn you both into glass cannons and not just you or the enemy. I like Call of Duty games for this, when the difficulty is ramped up, the enemies seem to take around the same amount of damage but your just about as brittle as they are. So your on an even keel.
Games where the upping of the difficulty does one of two things:
1) Turns the computer into a major cheating bastard
2) Makes the enemies just take a hell of a lot more damage
I tend to stay away from, the challenge just feels artificial.
The other games I only really play on easy mode are the Street Fighter games as I'm no good at them but like them anyway and RTS games which I also love but I'm no good at. Anything else I'll either tackle on normal or hard mode.