Well, I'd say give it a solid hour or two at least. That's my rule of thumb, at least. If in those first two hours the game fails to ensnare me with some aspect of it, I'm out.
For instance, I made it a little over an hour into Undertale before I gave up. I didn't hate it, it just wasn't doing anything for me. I didn't find it compelling, the gameplay wasn't fun, the characters were alright but unspectacular, the visual style was the part of retro that I don't care for. All in all, wasn't having fun, so I went away. But I gave it that first hour, just to make sure. I felt like quitting 15 minutes into it, but persevered, if for no other reason than to remove any doubts later on.