Three Four games spring to mind:
Ebonstar
I was young and I hadn't a clue what was going on, so I gave up. I returned to it recently, and realised there were controls to fire, as well as move (prior I thought all you could do was turn 360° on an axis). That explains everything! It's still hard as fuck, though.
seriously, fuck this game
Horror Zombies From the Crypt
...because I got stuck, and had no idea how to progress from the early part of the game, so my friend and I just gave up. Never played it again.
Was later informed that my friend figured it out, and found the rest of the game rather enjoyable
Psycho
Had no idea what was going on. Entered the mansion, ghost shows up. What next? This is crap! Gave up, never played again.
Although if someone knows what to do next, I'd be eternally grateful
Dr. Franken
I got stuck, and lost in the mansion, looking for a body part. I think I was 37% of the way through (that number has stuck in my mind). Annoyingly, every room respawned enemies, which was a pain in the neckbolts when you had to traverse from one end to the other, looking for the elusive piece of leg you were missing. Gave up, and like the tantrum-bucket I was, threw my Game Boy at the wall. Like all Nintendo hardware, it survived.
Looking for pieces of your lost love. You were made for each other!