Not usually. Only in a few special cases.
Call of Duty 4 - the mission "Heat". Yeah, fuck that mission. Reduced it to Hardened from Veteran. That mission was damn-near impossible to beat.
Mass Effect 2 - Was planning on doing a hardcore/insane run like I did in ME1, but the higher difficulties just made the already boring-as-hell gameplay even more boring. So I didn't do that.
Dragon Age Origins - I was just on Normal, but the fight with Branka was just ball-bustingly difficult. I made it even worse on myself since I didn't have a mage in the party, and not many health potions. Even on easy that battle was a huge pain in the ass.
Pertaining to Skyrim - a couple of times I've wanted to lower the difficulty, but I just left the dungeon/ruin/whatever, did other things, leveled up, then came back and stomped on everyone.