Let's see, besides the innumerable amount of Nes/SNes games I couldn't get through way back in the day...
FTL: Faster than Light - even when the RNG was on my side I've never beaten the Rebel Flagship, ended up losing interest in the game after the 10th or so loss to it.
Kerbal Space Program - I am just not good enough at mathematics and engineering to do anything in this game. Hell, I never landed on another planet before I quit.
Darkest Dungeon - was actually in the Darkest Dungeon, but a bad string of RNG ended up killing my entire party in a single fight and it would've taken me far too long to grind back up to that point.
Final Fantasy X - I can't remember the name of the boss, but there was a mid-late game boss who just shredded me because I'd never quite got a hang of the leveling system and was severely underpowered, but couldn't go and farm to make it any easier because of where the fight was taking place. A shame since I'd liked the game, but wasn't quite keen on restarting and retreading 40 hours of content to give the boss a proper go.
Lost Odyssey - Another farming problem with an RPG, except this time not even against a main boss. Just a mini-boss, out in a field, without any random encounters nearby. At first I thought I was supposed to lose the fight because of how hard it was destroying me, but no, turns out I forgot the golden rule of an RPG: Grind Whenever, Wherever, Until It Takes An Hour To Level Up Against What You're Fighting.
Mass Effect 2 Insanity Run - Fun fact: did you know the way Mass Effect 2 calculates enemy strength is based off your upgrades? And that if you do a new game+, and thus lose all your upgrades, the enemies will retain the strength scaling based off all your old-game upgrades because apparently it was programmed by a chimp? I got up to Garrus' recruitment mission and was getting ripped apart trying to close a shutter, and when I found out it was because I'd brought my level 30 along and the game was expecting them to have 60% more health and shields than they had, I just ended up quitting. Fuck that noise, I'll go fight the Nameless King in DS3 instead, at least he's fair.