Study Says Gamers Can't Tell "Adaptive AI" From a Placebo

PatrickJS

New member
Jun 8, 2015
409
0
0
Study Says Gamers Can't Tell "Adaptive AI" From a Placebo



A new study finds that gamers who are told they are up against an "adaptive artificial intelligence" get more challenge and enjoyment out of the game - even if it's all in their head.

Solo gamers know this scenario best - the moment when you've finally come across the perfect difficulty setting for the computer player. It's not too hard - nobody's getting any stat bonuses - nor is it too easy, with the AI opening their virtual gates before your invading armies. The ideal difficulty setting is one that keeps you on your toes, but never feels unfair, one way or another. But what if all that was just in your head? What if checking "hard" or "very hard" produced the same result - but you convinced yourself the game had become more challenging?

Those are the sort of questions being asked in a recently published study from the University of York. Paul Cairns and Alena Denisova asked 21 volunteers to play Klei Entertainment's indie survival simulator <a href="http://www.escapistmagazine.com/news/view/141028-Dont-Starve-Together-Given-Away-to-Dont-Starve-Owners
">Don't Starve. The first time, the players were told that the game's map would be randomly generated. During their second playthrough, however, the gamers were told the map was created on-the-fly by an "adaptive" artifical intelligence, which could change things based on the player's skill level.

After each round, the test subjects filled out a survey about their experience.

"The adaptive AI put me in a safer environment and seemed to present me with resources as needed," said one player. "It reduces the time of exploring the map, which makes the game more enjoyable," said another, and these responses really sum up the general attitude of all participants.

Of course, the effect was entirely in their heads. They played the same game both times, no adaptive computer "director" at all. When faced with a challenge in the second play through, they thought to themselves, "well, that clever AI is good at presenting me with scenarios attuned to my skill level!"

A later experiment with a larger group of volunteers and control group repeated the results.

This is certainly something I'm torn on - I would love to have developers take the placebo effect in mind when designing computer players or NPCs; and yet, I'd hate to be one of the suckers.

What games do you guys think could use the placebo effect to best results? My money is on Spy Party [http://www.escapistmagazine.com/news/view/124629-SpyParty-Declassifies-an-Open-Beta] - imagine the paranoia that could be induced if there was never any spy.

Source: New Scientist [https://www.newscientist.com/article/dn28319-placebo-effect-works-in-video-games-too/]



Permalink
 

Jadedvet

New member
Jul 1, 2013
48
0
0
Ah, the Placebo effect; we sure are good at finding things when we are told to look for them. This experiment seems a bit weak though. Having played Dont Starve, I think playing another two rounds would enlighten most of the players who were fooled. "Wait, it should be getting easier but there is nothing but wasteland and horror"

On the other side of that coin, real AI takes a bit of time to see as well. I played so much Left4Dead that its "AI director" became predictable in when it would turn up the heat.
 

chimeracreator

New member
Jun 15, 2009
300
0
0
The methodology they used for the research (as described here) only weakly supports their hypothesis. Since they always exposed players to the "adaptive AI" after they first played a "random map" the players were naturally more skilled and knew what to look out for when playing. So it's entirely possible that all the experiment showed was that players were projecting their increased performance at the game with a fictional adaptive AI.
 

Jubbert

New member
Apr 3, 2010
201
0
0
Wow, this whole "experiment" seems like complete bullshit. Not to mention that when people ask for adaptive and smart AI in games, they fucking mean that the AI in the game needs to be good - Not the damn map generator, who gives a fuck?
 

Squilookle

New member
Nov 6, 2008
3,584
0
0
Jadedvet said:
On the other side of that coin, real AI takes a bit of time to see as well. I played so much Left4Dead that its "AI director" became predictable in when it would turn up the heat.
Couldn't have said it better. AI and many other kinds of design and behind the scenes polish don't become apparent until playing the same thing many, many, many times over. Only then can you sort the cream from the crap.
 

Fat Hippo

Prepare to be Gnomed
Legacy
May 29, 2009
1,991
57
33
Gender
Gnomekin
Squilookle said:
Jadedvet said:
On the other side of that coin, real AI takes a bit of time to see as well. I played so much Left4Dead that its "AI director" became predictable in when it would turn up the heat.
Couldn't have said it better. AI and many other kinds of design and behind the scenes polish don't become apparent until playing the same thing many, many, many times over. Only then can you sort the cream from the crap.
Yeah, if I was playing an RTS with a braindead AI and was told it was actually "adaptive" I might believe them while playing the first round. But a dozen rounds later, I'd probably see through the bullshit.

This kind of reminds me of FEAR. People praised the AI soldiers for the flanking maneuvers at the time, but it didn't take long for people to realize that a lot of this flanking was actually pre-scripted behavior. Of course, since this was a single-player narrative experience, it wasn't necessarily a bad thing, since it still delivered an enjoyable experience. But in a game which is designed to be frequently replayed, and potentially even in a multiplayer setting? People will be intensely disappointed when the illusion is shattered, so it's probably a good idea to be upfront with them about the AI is (and isn't) doing from the start.
 

Strazdas

Robots will replace your job
May 28, 2011
8,407
0
0
a single round is never a good tests because it does not provide perspective. when you have nothing to compare it agiasnt you can make up all kinds of theories. also they havent even tested any AI to begin with. looks like both the researchers and participants didnt even knew the terminology.

Fat_Hippo said:
This kind of reminds me of FEAR. People praised the AI soldiers for the flanking maneuvers at the time, but it didn't take long for people to realize that a lot of this flanking was actually pre-scripted behavior. Of course, since this was a single-player narrative experience, it wasn't necessarily a bad thing, since it still delivered an enjoyable experience. But in a game which is designed to be frequently replayed, and potentially even in a multiplayer setting? People will be intensely disappointed when the illusion is shattered, so it's probably a good idea to be upfront with them about the AI is (and isn't) doing from the start.
back when first FEAR came out, flanking AI that seemed to react based on where you were trying to push (the enviroments were closed down enough for this illusion to work) was unprecedent. thats why people praised it. it was the first AI that used active cover (like a grenade flipped a table and AI would use it as a cover) and did something more than just sit around or run straight at you.
 

Chessrook44

Senior Member
Legacy
Feb 11, 2009
559
3
23
Country
United States
I'm now predicting that game companies will look at this, say their game has adaptive AI, but be lying in order to save money and cause the placebo effect. A year from now someone will catch EA doing this and a scandal will erupt for a month or two.
 

MHR

New member
Apr 3, 2010
939
0
0
What comes to mind is COD Ghosts "fish AI"

Fish swim away from you in real-time!

Which Mario 64 also has. It's been around forever.
 

mad825

New member
Mar 28, 2010
3,379
0
0
The only question remains; Was it the game or the researchers that convinced the participants who fell for the placbo? Did they actually do better or worse? For fucks sake, nothing is being measured here expect for people's thoughts.

Like any Rouge-lite game. You get good and bad runs, some runs will destine you to lose while in some you might might excel - it's random.
 

elvor0

New member
Sep 8, 2008
2,320
0
0
Jubbert said:
Wow, this whole "experiment" seems like complete bullshit. Not to mention that when people ask for adaptive and smart AI in games, they fucking mean that the AI in the game needs to be good - Not the damn map generator, who gives a fuck?
You're missing the point. In a game like Don't Starve, the map can be as much as an enemy as the actual enemies. Obviously it's important that that is handled with care too. Noones stating the the map generator is the ONLY important element of AI.

It's similar to the AI director in L4D or Amnesia. Despite the fact that the AI director and the enemy AI are different things, they are still inherently AIs. Yet the AI director for both Don't Starve and L4D perform the same task: adaptively alter things in a way defined by its programming to either increase or decrease the challenge for the players and to keep things different on repeated playthroughs.

Though I'm not going to argue with the experiment having issues, this is more discussing out your points about AI.
 

Lightknight

Mugwamp Supreme
Nov 26, 2008
4,860
0
0
This doesn't really establish that they cannot tell the difference between adaptive AI and a placebo. To test for that you'd need to have a group that is actually playing with adaptive AI.

This is like telling someone that good chocolate is actually swiss imported or that the cheddar they're eating is really from some exotic mountain village. They will enjoy the food item more even though it isn't actually the "real stuff" they think they're having.

But when comparing the "good chocolate" or "exotic cheddar" with chocolate/cheese that is actually hand crafted from some other location the difference is often noticeable.

So this study doesn't really tell us anything. It just confirms the principle of how to assign a more special designation to a product that is being consumed to enhance the experience. That's really old hat psychology.
 

Mikeybb

Nunc est Durandum
Aug 19, 2014
862
0
0
I'd like to think I wouldn't fall for it, but at the same time can't guarantee I wouldn't.

The only thing that sounded like a warning flag to me was that don't starve compiles the map prior to any gameplay, at least from what I assumed.
Whether I'd have picked up on that or not though is unknown.

Interesting study though, even if we don't know full details of the methodology.
 

snekadid

Lord of the Salt
Mar 29, 2012
711
0
0
Ug.... interesting study but they tampered with their results by.... MAKING THEM PLAY THE SAME GAME! Of course things seemed easier the second time when they knew more than nothing. The first time you play that game you're lucky to make it past day one because its so unfriendly and doesn't explain anything. It was a loaded test and its results are worthless.

note: I love dont starve, and dont starve together is amazing, but the game does hate you and wishes for your demise.
 

Weaver

Overcaffeinated
Apr 28, 2008
8,977
0
0
I mean, maybe the game just generated a more friendly map the second time.
 

Pinky's Brain

New member
Mar 2, 2011
290
0
0
Fat_Hippo said:
People praised the AI soldiers for the flanking maneuvers at the time, but it didn't take long for people to realize that a lot of this flanking was actually pre-scripted behavior.
The key to good "AI" isn't whether it's scripted or not, but how many scripts the developers wrote. Yes, the scripts should be able to use dynamic input rather than being pure set pieces and NPCs should be able to cancel out of them, but when all is said and done better "AI" comes down to more scripts.
 

Fat Hippo

Prepare to be Gnomed
Legacy
May 29, 2009
1,991
57
33
Gender
Gnomekin
Pinky said:
Fat_Hippo said:
People praised the AI soldiers for the flanking maneuvers at the time, but it didn't take long for people to realize that a lot of this flanking was actually pre-scripted behavior.
The key to good "AI" isn't whether it's scripted or not, but how many scripts the developers wrote. Yes, the scripts should be able to use dynamic input rather than being pure set pieces and NPCs should be able to cancel out of them, but when all is said and done better "AI" comes down to more scripts.
What I meant was, the soldiers were scripted to follow one specific set of actions, which they did in the same manner every time. They didn't change their pathing or take different cover dependent on the actions of the player, which would have been the dynamic version of the same AI, but also vastly more complicated, since it would need to read the actions of the player and interpret them in a meaningful manner and react appropriately, as opposed to simply following the more or less same script every time. So sure, all AI may be scripting, but there's a difference between having a single script which never changes, and an AI which actually adapts to the player. However, when done well enough, this single script may appear to be an adaptive AI, though only during the very first encounter, after which the illusion falls apart.
 

CaitSeith

Formely Gone Gonzo
Legacy
Jun 30, 2014
5,343
358
88
Lightknight said:
Weaver said:
I mean, maybe the game just generated a more friendly map the second time.
Or, maybe they were now more familiar with how to play the game?
Either way, it seems they fell for it.