Sorry if I repeat anything that's already been said, but hopefully I can add something useful.
I get the impression that it really has to do with targeting the largest possible demographics. Gamers, traditionally, have been white, male and relatively young. That is obviously beginning to change, and consequently we are starting to see more diversity in the casts of games, but until we get more and more socially-aware people into the industry, both in development and publishing, we're still going to have these problems.
The easiest solution, in my mind, is to simply start telling more linear stories. As games become a more capable space for narrative, it means we can pull this off far better than in the past. These days, "player choice" is a buzzword and a bullet point on the back of your box, and games rarely deliver in a satisfactory way on those promises. This fits into the larger trend of games trying to appeal to the widest demographic possible by having more and more features, game modes, options, etc., which is itself a byproduct of skyrocketing development costs; in many ways it's a bit of a vicious cycle.
I think we just need to realise that most games are not going to appeal to everyone, and move on. We can learn a lot from looking at PopCap and other casual-oriented developers, who tend to have games without meaningful human characters; the gameplay is simple, but they can also maximise their audience by focusing on presentation that appeals to everyone. Recently, a guy from Infinity Ward (whose name I forget) mentioned in a GameTrailers interview that they had realised a while ago they can't appeal to everyone with Call of Duty, and shouldn't even try, which I thought was a bit surprising for such a successful game, but also probably quite telling. By focusing on creating a quality core experience, developers can use their talents to make the best games in one genre rather than the worst game across five genres. Similarly, simulation and role-playing developers have known for years that they can be successful by appealing to a hardcore fanbase without trying to cater to other demographics.
I'd love to see more games that feature more diverse racial/ethnic groups, as well as more diverse gender and sexuality representations, especially in leading roles. A recent study, for example, found that there is not a single game on the market that has a Hispanic protagonist (excluding games with character creation features; such features aren't an acceptable compromise anyway). The assumption, as said above, is that the people playing are going to be white and male, so the white male is the most obvious choice for a leading role. Yet even in games that feature lead female characters, often those characters are highly sexualised, objectified, or not treated in a realistic way with respect to social issues (i.e. the woman you play as doesn't have to deal with making less money than a male with the same job, doesn't have to deal with sexual harassment, etc.). Of course, this is because we still assume that the target audience is males; the point of having a female character isn't necessarily to be proactive or to appeal to females, it's to give the male players some eye candy. This is evidenced by the fact that most games featuring female characters are third-person instead of first-person; the only exceptions I can think of are Portal (which barely even mentions Chell's gender to begin with), and Trespasser (an early example of "full body immersion" that was likely just an excuse to let the player look down the main character's shirt).
In any case, the industry and the players both need to acknowledge that not all games are for everyone, and that it can be an opportunity to appeal to minorities; the incessant pandering that makes up the games industry these days has to go if games really are going to be taken seriously in the future.
