Okay, to the OP:
I have some good news. If you want psychological horror try the below freeware game. Anything done by Yahtzee in his "Dafoe" series (7 Days a Skeptic, etc..) is liable to be good but White Chamber is something of a standout for the psychological horror adventure game genere (and trust me when I say psychological).
http://www.studiotrophis.com/
Not long, but you'll probably get a bit of enjoyment out of it (I've posted the link and recommendation here before).
I also have some "bad" news. Your best bet to find true psycological horror games is largely going to be to search for either independant productions (some fairly old) like the above, or to go looking around used game bins, abandonware sites, and the like. The reason being is that psychological horror as a genere is pretty much dead. At one time there was a ton of games all about "creepy" or "off" settings without being over the top "OMG" horror all the time, but in recent years it's died out. "Sanitarium", "I Have No Mouth But I Must Scream", and even "Dark Seed" (2 games) are all good, if dated examples.
Today pretty much all horror games are pretty over the top, and don't do "unsettling" as much as flat out "horrifying" in a very in your face fashion. There is nothing unsubtle or understated about the atmosphere in "Dead Space", Yahtzee called it the "USS Kill Beast Buffet" for a reason. In many other games your basically walking around waste deep and gore and corpses with stuff popping out to kill you every 15 seconds. This is true of everything including Silent Hill (which didn't impress you much, though I'm not sure which one you played).
Still for games that are out there the closest your going to come (and I recommend this by the psychological designation, not specifically firing a shotgun of horror games) are:
#1: Rule Of Rose, this game (contrary to what some have said) DOES make sense. Also once you figure it out you'll probably know why I said I considered the relationship with the dog in this game one of the most powerful person-animal relationships in a video game.
#2: I recommend "Silent Hill IV: The Room". Despite comments about your experiences with the series, this game is the "off child" of the bunch which was originally not developed as a Silent Hill game but later inserted into the series. It's the game that a lot of serious hard core Silent Hill nerds dislike because it was so differant.
It also did surrealistic horror better than anything I've seen so far in a game. Your playing a desperate guy trapped in his apartment for days who is running out of food who winds up basically trying to escape by crawling through holes in the wall into all kinds of truely messed up dreamscape enviroments. It's... differant. Though the mechanics for exorcising the ghosts do become a bit annoying.
#3: Fatal Frame is very atmospheric, and the second one (which my copy of is missing) is perhaps the best overall. However for psychological horror I recommend the third one as it's a bit differant from the others taking place under a veneer of normality with characters from the first two games touching base in an apartment they share. The first two are pretty much a lot of creepy rooms, shock moments, and ghost busting with a camera. The third one was not as popular as the first two it seemed, BUT it also broke that format and tried to go in a slightly differant direction than "alone and surrounded by overwhelming evil coming out of everywhere"!.
For the record I never did beat the first game (which I'm guessing is the one you played) as I couldn't figure out where to go in the Demon Tag chapter despite directions. I had that bloody respawning ghost showing up going "oh my eyes! oh my eyes!" being more annoying that anything while I looked for the mask. Dusted the bloody thing what was probably a hundred times and the figure "enough of this". So chances are I share your pain.
Not much new here, but basically the "current gen" of games has little in the way of horror titles. With the specter of censorship everywhere (whether you agree with me on it or not) it seems people just aren't risking making them or bringing them to the US. What ones there are, basically come down to a variation of "over the top alien/zombie shooting gallery".
There are good "monster bashing" games, some with very interesting atmosphere, but most are very direct as opposed to cereberal or psychological. Not much in the way of mystery or "veiled malevolence" everything is quite obviously malevolent in the most cheezy and over the top way for the most part. Even Condemned is pretty straightforward, though it does have a psychological horror element where they guy communes (literally) with his inner demon of alchoholism and later kills it in a bar fight (I kid you not).