I was not necessarily a kid then, but there was a castle in Alex Kidd in the Miracle World. I breezed through the rest of the game, but the music in that castle as well as the ramped up challenged freaked me out. The same for the ghost levels in one of those Mario games. Something about the music that put me off.
More recently, around ten years ago, playing the original Clocktower scared the piss out of me. And a few years ago when I played Zelda OOT for the first time, that as well. Everyone seems to fear the well of DOOM.