It does seem difficult to get a good, scary game out of a mainstream developer. But I think it could be possible. The crux of Yahtzee's argument, from what I could gather, seems to be that there's just so many details and effects crammed into a mainstream game it can't be subtle. I'm not so sure, but it would take a lot of finesse and careful planning.
To bring about good, tension-building you need very quiet moments, but that leaves the question of what the sound designers are supposed to do. Maybe if instead of putting most of their efforts into loud noises, they could work on softer, more stealthy sounds that would keep the player on edge; low creaking floorboards, muffled scrapes and shuffling, whispers, and so on.
Graphical subtlety would be more difficult, but I think doable. Now adding detail to something hideous and disturbing will help make it scary, sure, but you also have to build up suspense as well. You can't have endless hideousness or the player gets accustomed to it. So what else can the graphical designers focus on?
Well, when Yahtzee reviewed "Dead Space" he said it would work better if the ship had actually looked like a safe, secure place to work and live before the corpse-mutating aliens showed up. The graphical artists could put a lot of effort into making the game environments look as safe, inviting and secure as possible. That way when everything goes to hell, it will be all that more shocking that something that locations that seemed so reassuring and comfortable had turned into such nightmarish horrors.
Another subtle effect they might try to create is something to recreate glimpsing things out of the corner of your eye. We've all had that happen, when we've just barely seen something off to the side, but when we turned to look, it wasn't there. If graphics artists could find a way to reproduce THAT, then they'd have a great tool for making the players nervous and scared.