Not including the games I played at my friends' houses:
Runner's up is Star Wars Empire Strikes Back for Game boy, which I mainly couldn't finish because it had no save and took a day to play. I just never had the time to play it all through in one sitting.
However the real kicker is Lemmings Paintball. Particularly the level in Mayhem: Babylem 5. I recently found an abandonware version of this game and I still can't beat that level. It's ridiculously complicated: each switch controls at least one door or platform (which you can hear trigger) and a slippery pad (which you can't hear). Plus, half the switches are pressure pads which you can't see anyway, and half of those require being shot! It also has a time limit and your lemmings die in one hit. This is usually fine, but you really don't have the time to explore this level properly, as there are so many possible combinations of switches you need to figure out.