I'm opposite. The idea that the down was indifferent to me and just let me wallow in my broken psyche was very depressing (Silent Hill 2 made me cry quite a bit, actually), but I didn't find it especially scary.
In 3, I had the sense that the town actually wanted me dead, which made me actually feel a real threat. Now there was more enemies, there was blood pouring from the walls, there were people dying everywhere (not sure if for real, but that poor, poor shopkeeper...), and Silent Hill had actually leaked out and attacked the city you lived in specifically to get you. So I never thought it was "fight to the head demon and kill it" as much as I thought "OH GOD WHY HELP WHAT DID I DO AAAAUGH!"
This is why Silent Hill 4 is the scariest in my opinion: Because the town is following the lead of a man who's out to get you specifically. Best of both worlds.