I've been thinking about dialogue in games a lot lately.
To my mind, there are two kinds of ways to approach the player-character relationship -- for the purposes of discussion, we'll call them "play-acting", and "role-playing". In play-acting games, the character acts as a proxy for the player; Dovahkiin in Skyrim, or Shepherd in Mass Effect. You pick the choices you would pick, and the character acts as a tool which expresses your agency within the game-space.
Role-playing games, on the other hand, task you with stepping into the shoes of an already-established character, with his control scheme, and indeed the entire perceptual framework of the game, carefully crafted to make you feel like that character, and approach decision-making the way they would. Batman in the Arkham games, for example.
And what's interesting to me, and worthy of deeper exploration, is that the vast majority of dialogue in games has been of the "play-acting" variety -- EVEN in games where the player-character relationship is otherwise almost completely "role-playing".
Dialogue in "play-acting" games, of course, has to account for the full spectrum of player choice, from moralistic idealogue to dickass jerkface. The result is that you have to invent this wiggle-room which could theoretically allow for the main character's personality to veer wildly between two extremes. Even when it's smartly toned-down ("What if it's just good cop vs bad cop?") it still leaves the character feeling vaguely bland at the center.
While Deus Ex: HR does indeed suffer from that problem at a top-down design level, and Adam feels a little bland as a result, I will say it approached the actual meat-and-potatoes of dialogue in a very interesting way, and that's probably why the "social boss battles" got such recognition.
See, rather than giving you a cross-section of every conceivable response you could want to make at any given juncture (as the Mass Effect series does) it gave you a selection limited to what Adam Jensen would or could say, at any given time.
But more importantly, it uses that character's background as a way to inform your decision-making. So rather than simply blindly picking lines of prose, you've got that nice little "PRESS" "CHALLENGE" "INTERROGATE". Your potential choices are framed as tactical decisions, taking into account Adam's history as a police officer, in the context of an interrogation -- in THE EXACT SAME WAY that the Arkham games take into account Batman's skill-set, in the context of combat.
I'd like to see more along those lines. One can imagine a game where you would play an expert orator/logician/rhetorician, and the core gameplay would be nothing but dialogue, navigating the nuances of human communication itself.
EDIT: And I think 99% of the time, when the dialogue system in a game falls through and you find yourself frustrated, it's because the game has momentarily forgotten (or never figured out) whether its main character is a proxy for the player's will, or a character in his or her own right.