As a good friend of mine said, a good RPG needs to things.
1) A good battle system. It doesn't need to be overly complex, just good. Remember, the player will be fighting a LOOOT of battles. If they're not fun, they'll get tired of the game and maybe quit.
2) A good story, or at least good characters. The story is like the carrot on the stick, urging the player to keep playing so they can see what happens next. And the characters help draw them into the world and make them emotionally connect to it. Having a good story and/or characters can really pull an RPG forward.
Here are some examples to illustrate my point.
ex1: Breath of Death VII (on steam). This is a straight up super-retro RPG. The story is a bit lacking, but it's meant to be a silly humorous game, and it pulls that off well. And the characters are fun and quirky. As for the battle system? Damn good. It's super simple, but very effective. As a result, it's good all around.
ex2: Eternal Sonata. Ugh...The combat system is BRILLIANT. Turn based meets action based. And it's REALLY fun to play. .....But the story, characters and delivery are all terrible. The game takes ten minutes to slowly describe a town for you, when you can see it right there. It just isn't well told. As a result, most people I know played it for the battle system, and then gave up later because they felt no motivation to continue.
ex3: Earthbound (SNES). Simple battle system, but well balanced. Good, quirky story. Everything came together perfectly in that game.
ex4: Mass effect 2. We all know the action was good, so there. The story wasn't quite as good as ME1...but ME2 made up for it with really good characters. They managed to draw me into the world, and keep me interested.
Hopefully you found that helpful.
EDIT: Also, Random encounters are not usually looked upon that well, so if you ARE using it and not the earthbound/crono method of bumping into enemies to engage them...Try using a system that lowers the enemy encounter rate based on battles won, or based on party level. It'll allow the users to reach the level you want them to be at for each boss, and allow them to explore your dungeons more easily. NOTHING is more annoying than trying to explore a dungeon, only to get into a long battle every 4 steps (I'm looking at YOUUUUU Black Sigil!

)