I think the main problem is that we can't all be universally scary of something in particular. I mean, of course there's such things like fear of death but this isn't something that you can easily simulate.
Some people aren't afraid of the dark, so one major theme instantly drops out. See what I mean?
To further prove my point, let's pick a recent 'horror' game - Evil Within. To me, it isn't scary, it's a fun run'n'gun shooter with annoying moments that supposed to scare you. To others it's genuinely creepy and scary.
Another problem within the genre is that it became too reliant on jumpscares. It can startle, but that's all there's to it. Being surprised or startled isn't scary nor is it too hard to achieve. Creating an atmosphere of fear and terror is a very difficult task that very few can achieve. Making it believable is even harder.
Take Outlast for example. Sure, it can be scary at times. But more often than not it's annoying. Annoying because you can't defend yourself even if it's perfectly reasonable for you to be able to. The game itself shows that main character (in cutscenes) can dispatch of his pursuers, that they are not invincible (except one but that's spoilers). This ruins the immersion completely making it extremely annoying.
Well, I am aware that it's very difficult to create a good balance between action and horror and easy way is to take away means to protect yourself. But it's not working if there's no good reason behind that decision other than easy way out for game designers. In many cases that option breaks away the horror, i.e. Dead Space. Once you get your weapon you get completely numb to all attempts of the game at being scary. You dispatch zombies by the dozens and your backpack is full of medkits, ammo and shit. Why am I should be afraid? There are many of them but a little caution and they will not be able to even get close, let alone hit me.
There's another way of dealing with this as shown in Alien: Isolation and (surprise!) Resident Evil 4.
Alien manages this by giving you weapons but renders them useless - your enemy is invincible! Of course game doesn't tell you that so you can unload your shotgun to no avail, but that's the beauty of it! You are the victim in the slasher movie and villan (as he should be) shrugs off everything until the end. Yes this is a good way but also a little cheap.
That's where RE4 comes into play. Until you reach middle of the story game is pretty straightforward. You shoot them - they die. Some (Dr.Salvador) can take more punishment than others but that only means that you should shoot them more than usual. A little tense but nothing unusual. And then it hits the scene. THAT. ONE. MOB. Low-key creepy ambient music. You shoot him, blast his limbs off, but he keeps coming. Yes, he can be defeated but it's a little trick that needs to be uncovered first. And before that you're caught unguarded. That moment is genuinely scary, even if you have a grenade launcher.
And lastly, there's likes of System Shock, Silent Hill and Cry of Fear. They manage to be scary all the way through - even when you have a small arsenal of big guns.
So I'd say making a scary game is a very difficult business.
