It depends on the game.
Most of the time I don't give up so much as lose interest and go on to other things for long enough that going back to the previous game is a hassle in itself.
But when it does happen, it's usually dependent on the time when I determine I'm not enjoying myself and don't have the energy or determination to push past whatever is bugging me.
Pokemon Black/White, I disliked the UI, the region, the insistence on shoving the dialogue in the player's face every ten minutes, and most of the Pokemon designs for that generation, but the real turning point was at about the fifth or sixth Gym, maybe ten hours in or so, where I realized after having not played the game for a month or two that I had no idea where I was supposed to be going or what I should be doing.
Baldur's Gate, I just cannot get past the implementation of AD&D rules in a video game. Maybe it's because you start out at level 1, maybe it's because I have little experience with old-school western RPGs, but the combat in the two Baldur's Gate games starts out so inexorably boring and tedious and frustratingly random that I just can't be bothered getting to a point where the player's party is actually competent. Two hours into the first game I came across a normal traveling location between the starting area and a town down south that you're supposed to go to that took me the better part of another two hours and more save reloads than I ever had to make in the entirety of a Dragon Age: Origins playthrough just to pass. So after about my fiftieth death over the course of five hours, give or take, I decided I was done. I get the appeal, and still want to give something like Neverwinter Nights a better bash because I love Knights of the Old Republic myself and don't know why I can't get into the D&D games, but it just infuriates me.
Final Fantasy XIII, I talk about too much. Far too much. I've talked about it today, because threads about it are inexplicably popping up again. But I gave it just about twelve hours before I couldn't go any further, and then almost exactly three years later I tried to give it another go because I had a hankering to see if I could stomach it better with a different perspective. Short answer is, no I couldn't. Didn't even take half an hour before I had to turn it off again.