296: Show, Don't Tell

[Blaine Kyllo]

Show, Don't Tell

Real people don't just talk with their mouths - they use their faces, hands, and bodies to express themselves. Now, thanks to advances in motion capture performances, videogame characters are communicating in equally subtle ways.

Read Full Article

 

[erbkaiser]

Excellent article. I am right now playing Enslaved, and the full range of expressions of the characters feels so natural and real that at times it is like I am watching a full motion video rather than 3D renders. This is also where Heavy Rain (and, to a lesser extent its predecessor Fahrenheit) shines: the characters react like real people, so it is much easier to accept them as actors.
It is also where the otherwise excellent Mass Effect 2 falls flat for me: all the acting is done by voice and a select few face muscles, the rest of the bodies is unnaturally stiff. Oblivion/Fallout 3/NV are even worse, here the faces are the only parts of the characters that act.

I wonder if this is a part of the uncanny valley effect -- as models become more and more detailed, their unnatural movements or lack thereof become more noticeable, whereas in the "simpler" games of the past with far less detailed characters this just wasn't an issue.

 

[Spark Ignition]

Enslaved was really, really good as far as the acting went, if it did (for me) fall a little flat on actual gameplay. Personally I can't wait to see L.A. Noire, and the facial capture they've been telling us will actually make it possible to judge whether a CG character is lying or not.

And then I want Bioware to use it. I agree plenty with your theory, the latest Bioware games (and don't even get me started on Fallout 3 & NV) are really let down, despite some of the industry's best voice-acting and dialogue, by the uncanny-valley faces. Back when we were looking down on isometric Baldur's Gate letting our imagination do most of the work there was no issue with this, but the more videogames try to emulate reailty the more we're going to notice the differences.

That said, LA Noire people!

 

[redisforever]

Nice article, I did notice a spelling mistake though. On page 2, instead of Heavenly Sword, it says Heavenly SwordM.

 

[Dastardly]

Show, Don't Tell

Real people don't just talk with their mouths - they use their faces, hands, and bodies to express themselves. Now, thanks to advances in motion capture performances, videogame characters are communicating in equally subtle ways.

Read Full Article

These are the kinds of advances that can move gaming forward as an artistic medium. There are those who worry this will move gaming toward those perhaps-too-guided theatrical experiences (Heavy Rain is the current go-to example). And for now, they're a little correct. As we're experimenting with what's possible, we tend to focus a lot on the elements that put this technology to use. L.A. Noire might be just that kind of game--pushing the limits of what we can demonstrate, perhaps at the expense of fast-paced action a lot of the time. But if we let these sorts of experiments turn us off from the techniques, I think we're missing out on what impact these advances can eventually do for story.

Think of all the things a well-directed, well-acted, and well-shot movie doesn't have to explain to you, because you can see it in the details and expressions. Less time spent explaining is more time spent blowing things up, or whatever the case may be. The story can have the same depth with less direct exposition. These techniques can eventually give games that same benefit. As the character model and animation are able to deliver important information, the player is freed from unnecessary text or cutscenes--making animation more theatrical will allow the gameplay to be more... game-ical?

 

[ahiddenfigure]

That reminds me, I've been meaning to go and buy this game, but a lack of income has prevented it.

On topic: Great article, non-verbal communication is such a major - yet implicit - part of human interaction. It can say so much without "saying" anything at all, and yet we notice it only when it's missing. It's nice to see a better use for this advancement in technology for motion capture and graphics rendering than "more explosions" and allowing characters to have subtle movements without being deliberately exaggerated.

 

[warrenEBB]

neat article.
Was lightly baffled by the final paragraph though. wasn't expecting branching storylines to be the takeaway. if anything, i'd hope for the opposite...

Cameron and Jackson both showed "virtual camera experiments" on their DVD extras- where they walk around in the CGI scene and cover the action with a rig (which was covered in markers, being tracked just like the actors) (NaturalPoint recently released this experiment as an actual product. and CG society ran an interview with someone who used it on the latest halo trailer, to get both realistic closeup shakey-cam shots and wide floating helicopter shots from the same art elements in Maya).

I think the key takeaway is that : you can get a great performance once, then go back and explore it multiple times.

Soon, you won't have to worry about missing an actor's best take because you were setup for a wide or reverse shot. You'll be able to go through again and again, choosing which parts of the performers would be best emphasized for your ideal cut of the scene.

and this is what games will be able to do better than movies. you'll be able to play a game again, and receive a different cut of the same story and performance. You'll even be able to customize the cut to each individual viewer (longer takes for people who don't enjoy MTV editing, for example).

Imagine playing Enslaved again, but with more focus on pigsy, until it feels like the game is actually his story. or again from the robots' point of view. you'll be able to revisit the same actors and sets from different perspectives, because this costs very little to the game's graphical resources.

When you start talking about branching storylines, you're introducing the idea of the player becoming the director - and not many players are good directors. Most of us want to explore an experience, not be burdened with designing it.

 

[lightbombmike]

"... is about eye movements and expressions," says Antoniades. "That's not something you can do without performance capture."

Wow. Way to diminish everything the animators on your team do. To say that MoCap is a silver bullet, THE silver bullet for creating meaningful characters and moments is rather naive. It is a tool, not a finished product out of the box. It is comments like this one that make animators hate and fear mocap.

No matter how great the actor, if you don't have an animator that knows performance and the principles of animation, all you are going to get is visual oatmeal.

I love having mocap as a tool in my animation toolbox. But its statements like these that make me cringe to ever stick up for it, and make an honest conversation about its uses that much harder.

 

[vxicepickxv]

"... is about eye movements and expressions," says Antoniades. "That's not something you can do without performance capture."

Wow. Way to diminish everything the animators on your team do. To say that MoCap is a silver bullet, THE silver bullet for creating meaningful characters and moments is rather naive. It is a tool, not a finished product out of the box. It is comments like this one that make animators hate and fear mocap.

No matter how great the actor, if you don't have an animator that knows performance and the principles of animation, all you are going to get is visual oatmeal.

I love having mocap as a tool in my animation toolbox. But its statements like these that make me cringe to ever stick up for it, and make an honest conversation about its uses that much harder.

MoCap is just the quick and dirty way to get real emotions on something that has the approximate size and shape of a human. If you want to go for something with a very different type of face, then it's not going to do anything. It's also not really going to do you anywhere near as much good if you take away the face, while still using body language and speech.

 

[ThisNewGuy]

Aww, did not expect the article to turn into an in depth look at motion capture technology. Still, good articles are good.

 

[DirkBelig]

Good timing on this article because I just played it last week - I'm getting the feeling a LOT of people picked it up from the Kmart sale - and the story and performances were the best part. I was a big fan of Heavenly Sword as well - Nariko, babe, call me - and my interest in Fringe was less because of J.J. Abrams than Anna Torv, who had voiced Nariko. That the show turned out cool was just icing.

As more games take the live-action filmmaking approach to their cinematics - using all the actors physically performing the scenes to be captured like how Uncharted does - then stiffly animated scenes like those in Mass Effect will become less acceptable from AAA titles. Deus Ex was the best game with the worst animation. Whenever a conversation came along, the player was yanked from the first-person view to a third-person vantage point to watch JC Denton randomly spaz around. Awful.

 

[clockpenalty]

Video games are computer simulations.

If this were about some algorithmic method of generating realistic expressions I would be interested.

Perhaps some way to enable a mass effect character (for example) to emote properly merely by receiving simple inputs as cues.

Instead of depending on actors.

This is the of technology that the gaming industry really shouldn't waste money developing.

 

[Evil Tim]

Nice enough in third person. Yet in an FPS, we're still looking at the world through the hovering eyes of a man who has no eyesockets, hair, or even a nose, who goes through life with no method of acknowledging anything he sees but pointing a gun at it. People talk about how playing as Gordon Freeman immerses them but never think of why they can't see the frames of the glasses he's supposed to be wearing, why you never see his face shift as he reacts to things. You're a strange, sterile thing in a world full of realistically rendered people with expressions and emotions.

It's why physical interaction between NPCs and FPS player characters so seldom works. At a very basic level, they aren't there.

 

[asdfzxcv12]

"... is about eye movements and expressions," says Antoniades. "That's not something you can do without performance capture."

Wow. Way to diminish everything the animators on your team do. To say that MoCap is a silver bullet, THE silver bullet for creating meaningful characters and moments is rather naive. It is a tool, not a finished product out of the box. It is comments like this one that make animators hate and fear mocap.

No matter how great the actor, if you don't have an animator that knows performance and the principles of animation, all you are going to get is visual oatmeal.

I love having mocap as a tool in my animation toolbox. But its statements like these that make me cringe to ever stick up for it, and make an honest conversation about its uses that much harder.

Stop being so insecure. Nowhere did he say animators weren't important nor did he diminish their importance in any way. He's just a strong believer in PC bringing something extra to the whole process. Hopefully it becomes a standard in the industry.

 

[Mallefunction]

Probably why I loved this scene in Brotherhood (http://www.youtube.com/watch?v=w7i6UsHzv7o&feature=channel_video_title) Their faces, stances, and eyes really convey their friendship, worries, and sorrow (despite the limited interactions we see in the game).

 

[Tupolev]

Nice enough in third person. Yet in an FPS, we're still looking at the world through the hovering eyes of a man who has no eyesockets, hair, or even a nose, who goes through life with no method of acknowledging anything he sees but pointing a gun at it. People talk about how playing as Gordon Freeman immerses them but never think of why they can't see the frames of the glasses he's supposed to be wearing, why you never see his face shift as he reacts to things. You're a strange, sterile thing in a world full of realistically rendered people with expressions and emotions.

It's why physical interaction between NPCs and FPS player characters so seldom works. At a very basic level, they aren't there.

Actually, it's because they're too-much there. Sure, some physical aspects of the character are neglected (rendered glasses in particular would be annoying enough that they would actually be far more distracting than immersive), but ultimately the control is far more direct than on most 3rd-person control models.

The indirectness of 3rd-person control (and yes, it is less direct; the player character isn't even always looking in the same direction as the player, creating a huge disconnect in visual knowledge input right off the bat...) is what allows them interact; by not being immediately in the shoes of the character, it's not all that weird when they do stuff that you don't have completely precise control over.

It goes far deeper than visible physical interactions, too. The reason so many FPS player characters don't speak except in cutscenes is that the direct association of first-person view means that if they say anything, it had better match up with the mental state of the player, and it's hard to do that consistantly with procedural speech act usage.

It's why, in the Halo series, the Master Chief is almost always accompanied either directly or indirectly by something or someone who can provide occasional exposition; even if Cortana is in his helmet, at least it's not him talking. When Bungie decided to deviate in ODST and have player characters other than The Rookie speak in gameplay (saying things like "Changing mags!" occasionally), some players found it to be slightly distracting.
Heck, Rockslider, who could probably best be described as someone who goes around smashing things in the most stylish ways possible in Halo: Combat Evolved, pointed out in his Halo 2 "Dissapointment" list [http://www.badcyborg.net/Other/Halo2Disappointment.html#Melee_attacking] that one of his complaints with the new melee system was the randomization of melee animations, saying that it weakened "the sensation of control".

 

[beema]

The facial expressions were definitely one of the first things I noticed when I played the Enslaved demo. Really cool stuff that certainly gives the game much more emotional depth. Too bad the rest of the demo wasn't as compelling because I completely lost interest in it.

 

[Evil Tim]

Actually, it's because they're too-much there. Sure, some physical aspects of the character are neglected (rendered glasses in particular would be annoying enough that they would actually be far more distracting than immersive), but ultimately the control is far more direct than on most 3rd-person control models.

Really, they'd only be annoying because of the false perspective an FPS is forced to use; because most people don't own a gigantic convex television that takes up their entire range of vision, we're forced to have an artificially narrow view of the world that consists only of central vision, and sometimes not even all of that. Gordon's glasses would be no more or less annoying than real glasses if they could put them where real glasses would be in his view, but they can't.

It goes far deeper than visible physical interactions, too. The reason so many FPS player characters don't speak except in cutscenes is that the direct association of first-person view means that if they say anything, it had better match up with the mental state of the player, and it's hard to do that consistantly with procedural speech act usage.

Yeah, but the problem is if they say nothing at all, the game really isn't asking the player to immerse themselves in anything; rather, it just has a blank space in the world for them to stand in and not have their presence challenged. It's like those cardboard cutouts of strongmen at beaches; a child can stick their head through the hole and pretend they're the strongman, but in reality there's nothing there; they're just a child sticking their head through a piece of cardboard, not being asked to act like they think a strongman would while they do so. Similarly, the silent protagonist is a cop-out; rather than trying to create a convicing everyman for the player to emphasise with ("I want to be this person"), it creates a nobody who they don't have to empathise with. It's avoiding immersion entirely so it can't possibly break it.

This falls down when we're asked to care about the nobody; for example, Alyx Vance falling in love with a man who has the personality of a tree stump, or trying to make people care about Rabbit dying in Medal of Honor even though the only way you can remember which one Rabbit even was was that he hung around with one of the guys from ZZ Top. Wait, no, that was Deuce. See?

 

[Danceofmasks]

Well, it's a giant step to be sure, but ... cinematics can only go so far.

When emotional responses within body language happens during gameplay, in realtime, while interacting with objects in a sandbox environment, then I'll be happy.
Especially if the player character can also interact in realtime via (for example) hotkeyed speech.

Someday I wanna make a character say, "drop your weapon!" *cast warning spell* "do it!" "get down on the ground, now!" ... outside of scripted dialogue, in an elder scrolls type game .. and have varied, believable responses.
Someday.

 

[MattJones]

I think to main issue with motion (and performance) capture is that it?s not actually a time saver, which in a roundabout way is what it?s frequently pitched as, a way to do something that would take animators a great deal of time to accomplish.

In reality, motion capture data can take hours, even days to clean up depending on the length of the animation. I?ve seen cases where it actually took longer to smooth out captured movements then it would have to animate it from scratch.

As I?m not a character animator I can?t really comment but I know quite a few animators who simply prefer to animate things themselves even when they have the option of motion capture.