I'm going to bring up two issues:
#1: One problem with horror is that humans are adaptable, nothing remains scary forever. If something doesn't kill us, it makes us stronger in a literal sense. No matter how horrible a situation most people are going to adapt to it after a while and about the fifith time a monster shows up, people are going to be thinking more in terms of "what are we going to do about that thing" more than an "argggh! a monster" panic response.
This applies to games as well, it's ridiculous to assume that a character is going to continue to be scared beyond reason by the same things again and again, and not adapt somewhat to their situation and the realities of survival.
This is also one of the reasons why typically you see horror in stand alone novels, or movies, as opposed to like 300 episode ongoing series, and why when someone does tend to draw it out it turns more into fantasy than horror because realistically the same variables are not going to generate absolute terror for the same people.
I think that most video games tend to get it right for the most part, with the protaganist becoming better adapted and more formidible and in control of the situation the longer he survives.
#2: When it comes to mechanics like "horror factor" or "sanity loss" in games, it's a throw back to paper and pencil RPGs. It's important to note that the role-playing is about taking control of a character with capabilities far differant from your own. Any kind of a game includes a degree of detachment from the protaganists, and when your playing a character far differant from yourself the detachment is even further. Nothing gets around the fact that we, the gamer, are sitting around a table playing a horror game with our nerd friends, or just made the desician to plop a horror title into our game machine.
A resistance to fear, and grip on sanity (in the short term) represet things that are going to apply the characters in the game which might have quite a differant threshold than the player. Someone who is a Marine, or career Navy (to use examples of people I've played with) might be quite able to deal with a lot of the situations in a horror game, like oh say finding a dead body, or losing a comrade. However if the character they are playing is a timid co-ed who acts as professor's research assistance with no combat training, or years of military experience the same logic might not apply. The mechanics represent the character's capabilities to deal with such things rather than the players. In the other direction a timid co-ed research assistant playing a Marine who has done multiple tours in Iraq, or a Naval veteran who has dealt with pirates on the African coast might respond to things in a very "meh" fashion despite how the player might react.
It's important to note that in the scope of computer games, the same basic logic applies. The character your controlling is not literally you, and is not detached from what is going on. For the person playing the game, the experience is no worse than a thrill ride or haunted house at the worst. On the other hand the protaganist in the actual game does not have that perspective. What's more the character your playing, especially in an adventure-style game, might be an even bigger nerd conceptually than the nerd liable to be playing the game.
Simply put a video game is never going to really scare a person playing it, except maybe some jump out shock, or being a bit creepy in a "WTF was the person programming this thinking" fashion, especially seeing as you knew ahead of time this was going to be horror. The nature of the medium, and choosing to expose oneself to horror invalidates that. Thus the need for RPG type mechanics taken right from tabletop play to give the experience integrity.
Perhaps with greater strides towards VR we will see a situation where games will become immersive enough to do better, by being able to literally involve the people playing as themselves. However the desician being made to experience a horror game is by it's nature ALWAYS going to dampen the experience.
Short of some kind of experiment, or hacker psycho who breaks into people's VR systems, traps them there, and runs horror game programs, I doubt we will ever see a genuine horror experience in gaming because there is ALWAYS going to be the seperation of the person knowing it's still a game, even then. As such, things like sanity meters and the like are probably going to remain a staple to anyone who wants to include such things.
#3: The original Cthulhu mythos stories by HP Lovecraft himself have not aged all that well, however some of the stories involving the mythos that he edited have fared a little better. He was so inspirational that as things stand now, people have simply done the same stuff so much better (even using his ideas) where his writing seems quaint in comparison.
I also do not think people are cynical enough to think of themselves as some kind of mould on a rock drifing through space to be honest, but given that space aliens have become common fodder for collective fantasies, ideas like that just aren't as shocking as they once were.
To put things into perspective the nerds of the 1920s didn't spend a lot of time drooling over the idea of lesbian sex with races like the Asari in "Mass Effect 2". The attitude was entirely differant.
Also "The Great Race Of Yith" was not malevolent or meant to be scary in of themselves. The point was that they were alien enough to not mind messing with the occasional lower life form (well from their perspective) in the pursuit of knowlege, but it's also noteworthy that at the same time that same group of beings arguably step up to the plate for humanity by providing crucial knowlege or pieces of a puzzle. After all they were the ones who fought the original war with Cthulhu and the gods he worshipped, and they who apparently placed things like the Elder Seals, before themselves being run off.
I think the attitude is sort of like how a guy might really like animals like alligators or something like that. He occasionally kills one to skin it and make stuff out of it, but he also does things like help propagate them, see to their protection, and defend their enviroment. It's not a perfect analogy, but Yithian intervention could be seen sort of like a guy killing an animal in your herd for his own benefit, but also chasing away your predators and arguably doing more good for you than bad, even if the end he doesn't view you as anything close to an equal.
Of course part of the problem with Lovecraft's work is that when he wrote that stuff he kind of felt humanity had peaked technologically. Right now we have weapons and things that go beyond what the Yithians ever had (albiet we do not have mental time travel). Writers from Chaosium used to make jokes like "what would happen if you nuked Cthulhu? He'd reform 15 minutes later... radioactive and mad" when by the actual concept Cthulhu himself wasn't all that (he isn't even actually a god, but a high priest and sorceror) a couple of nukes could probably solve the biggest problems in those stories. But the horror of the time was that by HP's logic nothing like a nuke could ever exist, he couldn't conceive of such a weapon, and he thought humanity would never progress much beyond the level of the weapons we had in his day. It's interesting when you consider that today we could easily massacre the most horrendously overpowering things that the minds 90 or so years ago could conceive to scare people. Says a lot about how far people can come when we become stronger than doomsday no-win scenarios we create for ourselves.
