Vulnerability is key, in my book. Loud screams and jump scares only go so far. Silent Hill, Clock Tower, Amnesia, and Pandorum all leave the player almost completely helpless aside from their own quick wits.
Isolation helps, but even with multiplayer you can get the players nervous by keeping...