You made mention of it, but more often then not, it really is the controller that turns many prospective gamers off from bothering. I know a bunch of non-gamers, and for the most part, they understand what needs to be done in the game, its just getting there that is the problem.
This is why the Wii has been a big hit, people pick up the controller and go "Ooo this is easy! All I have to do is turn this way!" Or whatever it is. When I handed my girlfriend the xbox 360 controller so she could play Call of Duty 4 for the first time, she looked at me like I had 3 heads. "This stick is to turn the camera, this stick is to move." But, on screen, she would just strafe and walk through doorways without turning the camera and inevitably die.
However, she got me Lego:Indiana Jones, and is an ace at that game. The camera is fixed for the most part, the character moves with one stick, he jumps with the A button, attacks with X, and uses an item (the off chance you get one) with B. Thats all there is to it. 4 buttons. Granted, it was co-op and I could finally play with her, but if left up to her, she is more then capable playing by herself.
I will always find the controller to be the biggest roadblock for many newcomers. The other stuff comes with time. Besides, what happens when that other stuff isn't there anymore to help them out?