Well, I must say I am a Big fan of Horror Videogames and I have some ideas why some games are very effective:
-The element of the unknown.
The Amnesia use this element very well. You don't know nothing about the monster that chasing you or even it is a monster in the game at all.
The game tries to "built" the right atmosphere and give you some clues of wha possible danger have to face [by reading the letters and having your flashbacks].
Then, even you finally meet the "monster", you can't see it cause your insanity prevent you to see it clearly.
For the Amnesia this method was very effective.
Another example is from the PC Adventure game "Scratches". Same formula: You don't know the danger you have to face, the atmosphere the game built and the small parts of the story you learn while you make a progress in the game.
In other words, this method, the "Fear for the Unknown", is one of the best methods you make you break inside you.
-The shocking element of a story.
Another important element for a Horror game. The story of the game it have to tell.
I must admit, I didn't liked so much the story of Amnesia, but because it have the element of the unknown handled it right, I enjoyed the game.
There are some strong examples I can tell, but I will tell two games to describe to different ways of story telling:
Silent Hill 2 and Saya no Uta.
Silent Hill 2 use very effectivly many different elements built up a mystery story inside the game like the dead wife of the our hero, the mysterious Blond girl, the Pyramid Head, the whole town itself etc, only for one purpouse: To learn the truth.
Many gaves follow this formula, but Silent Hill 2 done a very damn good job to make the this reveal as shocking as possible.
Saya no Uta....I will tell this right now. It is a Hentai game. A visual novel that tell a story with some erotic scenes on it. BUT with said, this game have a very, VERY shocking and disturbing story to tell. I really believe The recent games like Silent Hill 5 and Resident Evil 5 must learn a lesson and follow the steps of Saya no Uta to become again horror games. AND I AM NOT JOKING.
This visual novel have very strong writing and cause of that this games work really well.
As I said before in other threads, a game doesn't need to have something scary or a monster to scare you to death for you to feel "Horror". I am honestly disappointing that most people think like that.
To say the truth, many well known games, movies etc. use monsters, creepy places etc. to scare the people. And I respect that, BUT is doesn't mean because you had seen/played these movies/games many times, make more "tough" from Horror media.
Myself I played a lot of Horror games and do you know what makes a good Horror game?
If the game make you care about the protagonist of the story
If a game make you to care about the hero or other peoples if something bad happen to them, THAT IS REAL HORROR!!!!!!!!!!!!! And when I mean to care, I don't mean like in Dead Space or Resident Evil 5 b*llshit. You must be afraid the fact that maybe this character will die because you learn about this character what kind of people he is, what kind of motives he got etc...
This Kind of Horror I like the most inside a game.
THAT why I love Saya No Uta!!!!
And that some things I know for Horror games I enjoy the most.
I hope you I helped.