Honestly, I dont think it's "challenge" thats missing from today's games. The problem is "filler" and "fluff"... like you said, Arbitrary tasks.
Take God of War for example.
It's a great series with a magnificent storyline. Sure, it can get challenging, and if you don't make it to a checkpoint, you gotta start all over again, but you still have fun running through just trying to accomplish that task and when you beat the game you feel you EARNED it, especially since things get ramped up to "vicious" near the end.
Now the problem begins when fans REACH that end and look at how long it took them to do it... anything below 6 hours and people start bitching. And that is the ultimate source of gaming woes today... the "time" vs. "quality" conundrum.
You see, many games have decent enough storylines and riveting game play. But because the populace seems to get sand in their collective Waaah-ginas when ever a game falls short of the 40+ hour mark, game developers feel pressured to somehow extend that gameplay. For some, it's as easy a decision as tacking on higher difficulty levels, others satisfy this with various degrees of "multiplayer" or even DLC. But TOO MANY developers are falling back on the old RPG tactic of "menial tasks" to "extend" game life... such fun distractions as "timed races", "fetch quests", and "collecting hidden stuff scattered around the environment". Then Microsoft hit upon a wise idea... trot out this completely meaningless "rewards" "Achievements" system. Not to be outdone, every other console started eagerly developing a similar system, giving you online trophies and gamer points that earn you absolutely jack shit beyond the self-bestowed pride of telling your friends about it. It's the modern day equivalent of putting you initials in after a game at the arcade, only without the deviant joy you obtained when you put in "ASS" or "FUC"
Grand theft auto is my favorite game to roll out when I get on this soapbox about game extention through mundane tasks. I actually was really interested in the gritty crime story of GTA IV, but I have YET to finish the game because I swear it seems to just go on forever and ever and ever. This is made worse with the absolutely frustrating mechanic of "virtual friends" calling me every friggin 20 minutes wanting to go out on some sort of useless "date" that feels like it drags me away from the story and causes my game play experience to hit a MASSIVE trainwreck.
"fetch quests" and "escort quests" round out the arbitrary task tree and to me are game killers. I have many, MANY games sitting on my shelves where the world hangs in the balance, where evil still reigns, because somewhere along the way in my quest to save the universe from God knows what, I simply got BORED and couldn't bring myself to pick it up again. My heart goes out to those little virtual world's facing impending doom in the Limbo of "Hyena's Incomplete Games", because what started out as fun usually ended up as "work", and ultimately ended up collecting DUST.
I understand that people want to get what they pay for, I really do. Times are hard, and spending $60 bucks on a fly by night adventure, such as Terminator's 3 hour slap to the face makes gamers feel like they got ripped off. But somewhere, there is going to have to be a little give and take.. Gamers are going to have to stop whining and demanding two week long epics while expecting absolutely the best gaming experience ever, and Developers with really good games are either going to have to stop having a standard "industry" price for games or else find better ways of maximizing game time without using by-the-book fillers. I have long felt it's time that gaming publishers stop throwing an arbitrary $60 price tag on a game simply because every OTHER game on that console is $60. Games should be like books, or other products and decide individually what to price a game. Games like Metal Gear Solid 4 have EARNED the right to demand $60 bucks... new IP's like Prototype should have to work their way up, unless it's from a really good company that can command a slightly higher price based on "name brand recognition".
I hope I have given you food for thought.