Well, it's a combination of things and it's a balancing act.
One of contributing factors is a human character. That doesn't necessarily mean their species has to be human, but they have to be relate-able. Let us see their doubts, their dream, their moment of weakness. If the protagonist is married and gets propositioned by beautiful woman, let us see his internal struggle. It doesn't have to be a long struggle; it can even be very one sided, but let us see his moment of temptation, however brief. If you don't it causes a break with the character because they're acting in a way people can't relate to.
On the other hand, they need to have some redeeming qualities, or the audience will begin to dislike them. I read a book once that had a central protagonist and two "almost protagonists." The book tried to portray the story in a realistic light and to have the central character be very human. The problem was they were too human. They were selfish, and stoic, and apathetic. And they were unlikeable. At the end of the story one of them dies, and I felt absolutely nothing. I couldn't connect with any of them because they basically came across as a-holes.
The balancing act comes from the fact that, while a character has to be human and yet likeable, they also have to be consistent. Don't have them act one way to humanize them and then act a completely different way to redeem them. Their strengths and weaknesses have to be different (unless a single trait can be both a strength and a weakness of course). A perfect example I have with this is one book where a guy come across a succubus. At first seeing the creature, the man is overwhelmed with desire, barely holding himself back from her. And then... not even a page later he gets into a stare-down with it to show that it has no power over him and forces the creature to look away first.
WTF? You just told us that he was immensely drawn to the thing and now you tell us he's not drawn to it at all? The argument could be made that it was an example of that inner struggle i talked about earlier, and it could have been done that way, but it wasn't. There was no transitional phase where he came to grips with his desire. One moment he was so overwhelmed he could barely think, the next, nothing.
A great series to read if you want to learn more about characterization is Terry Goodkind's Sword of Truth series. Goodkind does a fantastic job of developing characters, but has a problem with them the longer they hang around. He has a very hard time keeping people relate-able. If you get a chance, read through his books and keep track of when you stop caring about certain characters and why. About midway through the series, Book eight to be precise, it goes so far that you almost lose your ability to care about the main character. Thankfully he rescues the character in book nine (while simultaneously blowing the rest of the series to crap).
In particular pay attention to the character Nicci. She's introduced in the second book but doesn't become a major player until book six. Nicci is one of my favorite characters of all time and easily one of the best female leads in any medium. She's flawed, and tormented, she doubts herself, yet she's very powerful, strong and very likeable.
In fact, in the last few books I found myself caring about her a lot more than I cared about the intended female lead Kahlan, who starts the series out strong, yet ends up being mostly inconsistent, largely due to the fact that she keeps getting kidnapped and almost raped so many times that it loses it's threat. She's almost raped in the first book, the second book, the fourth book, the fifth book, the eight book, then repeatedly in the tenth and eleventh. The first time it's an intensely emotional and horrific scene; the kind that twists your stomach in fear for the character and leaves you sighing in relief when they get out of it. By the time it happens in the last book, it fails to evoke any response because we've been here so many times that we already know it's not going to happen.
Which is another literary point to make. Keep dangers fresh. Don't use the same threat over and over, because eventually it loses it's tension if overused, and tension is the entire point of danger in stories.
Sorry, went on a bit of a ramble there.
