Often, it's when it stops being fun in anyway. If the story isn't fun to follow at all, that can hurt the game, but gameplay might make up for it (though truly bad ones, or completely absent ones can cripple a game).
What's really been getting to me these days though is the sin of not telling me something important. Look, I know there's such a thing as telling too much, but when I have to actually google the item in question, you've failed in an aspect of the game (this is somewhat forgivable, unless you hit Brutal Legend magnitudes).
I will temporarily stop a game if things get too challenging, for the sake of my controllers, though only one game in recent memory has gotten anywhere near that point (Demon's Souls). Often I'll pick up playing later for the satisfaction of beating that challenge (though I want to point out that with Demon's Souls, I was given an item, a message, a trophy, and a new and interesting local to visit next, meaning that the reward felt equal to the challenge. If my accomplishments were just given a shrug and a half hearted "good job" I would have given up completely. Remember that the accomplishment should be powerful enough to motivate the player on).
I haven't run into unbearable voice acting that much, and so long as at most one character is the problem, I can handle it. Graphics I only really notice when I'm given time to rest, and haven't ruined any game yet. Yes it does take away a bit, but I'm mature enough to have fun with game regardless (part of the reason I'm still able to play PS1 games, though maybe the brillaintly done backgrounds of some of the fixed camera angle ones contributed to that).