What I find the most scary in games is something that should be normal and mundane given a slight tweak so that it's not quite right. For example, one of the bonus missions in Splinter Cell: Double Agent has a room that is, for no discernible reason, upside-down. Especially in a game that presents itself as a realistic shooter, this is really disturbing. A similar moment happened in Thief: The Dark Project, where you're sent in to burgle some random guys house, and then you notice there are upside-down rooms, trees growing out of the wall, and other unexplained weirdness. This, to me, is a lot more psychologically scary than jump-scares or anything like that. You go into a situation with the expectation that it's normal, and then skew them to be weird.
Of course, therein lies a problem. Most people go into a horror game expecting monsters, a creepy atmosphere, etc. The key would have to lie in subverting expectations. Make it so that when they think they're safe, they're actually in danger. Maybe have a cutscene where the player thinks they're going to get the standard exposition and mission briefing, but half-way through, the person giving the briefing starts to change appearance slightly, or their mouth stops moving, but the dialogue continues. Just an idea.
Of course, therein lies a problem. Most people go into a horror game expecting monsters, a creepy atmosphere, etc. The key would have to lie in subverting expectations. Make it so that when they think they're safe, they're actually in danger. Maybe have a cutscene where the player thinks they're going to get the standard exposition and mission briefing, but half-way through, the person giving the briefing starts to change appearance slightly, or their mouth stops moving, but the dialogue continues. Just an idea.