Gamewise(not completely sure reality wise), Deep water and Fog: who knows what lurks in there? what's worse is when you think that you've seen something, then it disappears, then it rears up right in your face...
Dark, abandoned mines,caves, and chasms. In Morrowind, I went into one where the only things I encountered were rats and precious ebony. After loading up with as much of it as I could carry, I looked at the map and saw a passage I hadn't explored yet, and, thinking that it was a shortcut out of there, I went for it. Then I realized I was going deeper, and that the hallway was well-lit with torch light. And then I realized that the source wasn't benign torches, but a tall figure,wreathed in fire, storming towards me. I died panicking...
Being resolutely alone also doesn't help, especially when you're used to working in groups. Republic Commando's first RAS Prosecutor mission comes to mind, when you've spent roughly half an hour wandering the bowels of a ghost ship, seeing and hearing your comrades picked off by machines and Trandoshans, seeing the corpses of the ships crew strewn through the halls, not encountering any serious opposition all the while, scares the hell out of me. Then , when you see an organized squad driving back the Trandos, you feel hope... Which is abruptly stifled when you're forced to watch the Trando officer mow them all down in seconds, then turn to you...
And sound. On Kasshhyykk in KOTOR, you stop to listen for a second, and you hear loud footsteps and a low growling... Yes, it's only the background noise, but it's very unnerving...
Finally(and this is more an "Ohhhh Hell" kind of scary), in Total War games, watching a huge enemy army off in the distance, thinking, "Ah, he's going to keep lumbering away in the mountains, routine movements." Then, upon hitting the Turn button, seeing him march through a shortcut right up to the gates of your city. kind of a dawning comprehension moment...