While you were talking about the all-text, BioWare style approach, you said that the problem with it, and the reason most games don't implement it, is that it takes a lot of time and effort to implement in a game. This is definitely true, but I can't help feeling that the system you suggested would take just as much effort to develop. Adding in enemy morale isn't particularly difficult, but programming in all the context-sensitive dialogue options and the effect they have on enemies would, I assume, be quite difficult.
Most action games have dialogue anyway. In a lot of FPSes, the enemies shout at you in an attempt at intimidation, or yell out in terror when you're defeating them, and in many games your allies do the same. I think, honestly, that this is enough in most action games. SWAT is a different issue, being a tactical FPS, but in fast-paced games like Halo, most people would either forget to press the dialogue button, or ignore it altogether because it's easier just to shoot.
It's very interesting as a potential mechanic, don't get me wrong. I just can't help feeling that it's a largely unnecessary feature, one that is likely to keep the game in development for significantly longer, and largely be bypassed by the people playing the game. Like you said, enemy and friendly morale has already been implemented in many games. In Halo the grunts are easily panicked and run away; in Jedi Outcast the stormtroopers tend to retreat if you kill their officers; in Baldur's Gate enemies and allies can fail a morale check and break off the fight if things are going badly for them.
As interesting as a context-sensitive dialogue option would be, I'm just not sure it's necessary. Dialogue-heavy games like the ones BioWare makes are always going to be that way, and games that light on dialogue are going to stay that way as well. This doesn't necessarily detract anything from the game; it's just a different feature of that game type. I just think the existing systems in action games are enough, and the feature you suggested would mostly either be forgotten about by players, or outright ignored so they can spend more time killing the enemies.
In the example you gave, in the shoot-out with the gang of thugs, it's all well and good making one of them a coward, but unless you wanted to keep the game in development for ages, this would have to be procedurally-generated - assuming that this sort of thing could happen with every group of enemies - which would make things very unpredictable and could, in theory, end up with all of them being cowards. In your example, the player may well notice that one of the thugs is a coward, but is it easier to try and coerce him into retreating than it is to just kill him? Especially considering that killing the others would likely cause him to retreat anyway, I'm just not sure it would add that much to the game.