159: Button-Mashing Monkeys

Alan Au

New member
Mar 8, 2007
61
0
0
Button-Mashing Monkeys

"Testing unreleased games is both fascinating and mind-numbingly boring."

"For some, it's the ultimate dream job. Many aspiring game developers see testing as a way to get a foot in the door, a stepping stone on the career path to a 'real' game development position. But others have a much more dismal view of game testing, envisioning it as little more than a room full of button-mashing monkeys. Even so, people continue to flock toward the game testing labs at Nintendo of America (NOA) and Microsoft Game Studios (MGS), eager to live their own small part of the game development dream."

Read Full Article
 

Eagle Est1986

That One Guy
Nov 21, 2007
1,976
0
0
Very informative, I always assumed that each developer had their own testers though, it never occured to me that Microsoft would have their own testers for games, as they don't develope their own games.
 

tendo82

Uncanny Valley Cave Dweller
Nov 30, 2007
1,283
0
0
Not gonna lie, game testing sounds awful. Considering most of the job is bug hunting, I'm surprised portions of the testing process haven't been outsourced to a country with lower pay scales.
 

TheKbob

New member
Jul 15, 2008
367
0
0
Thank god Im getting a degree and approaching the hardware side. It has its issues, as well, but yuck on testing.
 

manicfoot

New member
Apr 16, 2008
642
0
0
I did some game testing back in December. The game was an FPS and it wasn't really fun. We did multiplayer matches for an hour a day which was alright.. But apart from that we were walking around maps bug hunting.
The frame rate was all over the place as well which got quite nauseating after a few hours. Still, we had massive COD4 LAN matches during lunch :D
Now I'm testing mobile phones instead and I have a lot more job variety now thank God.
 

shMerker

New member
Oct 24, 2007
263
0
0
The article pretty closely mirrors my own experience working as a tester on Perfect Dark Zero at Microsoft's Samammish campus. One thing I'd like to add is that the atmosphere was really a lot of fun. It was four guys to a cubicle and you'd have another team right behind you so there was constant chatter about whatever anyone wanted to talk about. That was a big part of what made it bearable once we started working 65+ hour weeks to get the game ready to ship. One thing that's funny about the fatigue is that it destroys the quality of your work because you're a lot less likely to notice any problems, but it doesn't stop you from rocking at the game. There were nights when I'd be completely out of it by 10 or 11 (which means I was totally useless for noticing or reporting any bugs) but I'd still be pretty high on the kill board. (these were balance and load tests, so we were supposed to be playing the game rather than checking for holes in geometry or missing textures or anything like that)
 

brazuca

New member
Jun 11, 2008
275
0
0
Maaan I had a completely different idea for gaming test. I thought that the designers (not the whole team off course) stayed with testers for some time to see their reaction. Also I imagine that there were a psicological team to hear reports of how they feel after playing 60 hours of games some games, like FPS or GTA stuff.
 

Necrohydra

New member
Jan 18, 2008
223
0
0
TheKbob said:
Thank god Im getting a degree and approaching the hardware side. It has its issues, as well, but yuck on testing.
If you think that hardware doesn't have its own testing, better think again :p.

Thinking about it, besides the obvious reasons like increased pay, I wonder why coding experience isn't required of all games' testers. If all these guys are doing is pointing out the bugs/problems in code, things may be expedited in the design process if the testers, knowing something about code, could provide better details on a bug than "You can wall clip on level 5". Heck, even being able to fix the simpler bugs in the code they're testing could give the game designers more time to tackle the tougher bugs/balance, or to implement more features...or a normal work week.

Just my 2 cents.
 

brazuca

New member
Jun 11, 2008
275
0
0
Yeah! And game testers would cost more. Easier to enslave the game designer.
 

PEWPEWGreenLaser

New member
Jul 23, 2008
45
0
0
Necrohydra said:
Thinking about it, besides the obvious reasons like increased pay, I wonder why coding experience isn't required of all games' testers...
That is a surprisingly complicated question and the answer will necessarily vary from publisher to publisher. Part of it is money, part of it is politics, part of it is logistics.

In general, there are two types of testers: Black box and White box. You're talking about white box testers (testers that actually break in to the code, do more granular debugging and actually fix 'easy' issues.) Black box testers test from a player's perspective and very few ever get deeper than poking through .ini's or game logs for specific information. Most testers in the games industry are strictly black box. There's a few good reasons for that and I'll try to cover a few of them.

As noted in the article, Microsoft does look to promote testers with coding experience. But as you mention, cost is a major concern for making this a requirement. According to Game Developer Magazine's 2007 Salary Survey, Quality Assurance Testers with less than 3 years of experience earn an average of $39,063 (which feels a little high, honestly, and is definitely a piece of information I need to take to my next performance review...), while your average programmer makes somewhere in the neighborhood of $83,383. Your average Designer (the third lowest on the survey) earns around $63,649. (The true travesty in the survey is that Business and Marketing are the highest compensated persons in the industry, but that's another subject entirely.)

As mentioned in the article, publishers do hire testers that can code. However, they're primarily involved in building testing tools and automation. They're also paid a hell of a lot more than your average black box tester, so there are usually only a small handful per game team (sometimes across an entire QA Department.) Generating and debugging these tools tends to take up a HUGE amount of time for these people. So actually getting their hands dirty and black box testing the product is virtually impossible. These Test Engineers are greatly appreciated (though usually silently) by the rank-and-file. Test automation usually kills a large amount of the drudgery in testing a title. Unfortunately, some tasks can't be completely automated, so you have to pay someone to sit around and watch things. Black box tester to the rescue.

With the growing complexity of console titles and the surge in MMO production (arguably some of the most complicated software systems on the planet), there is a HUGE need to test these things at run time. The simple fact remains that you just don't know what's going to happen until you turn the key. If you can find me a coder that can produce 100% error free code, 100% of the time, I'm going to start praying to every God known to man because the end times are nigh.

One of the other issues with hiring exclusively test engineers can be in the structure of the publisher itself. If "testers" are fixing bugs, then why aren't they on the production team? It mostly comes down to money here too, but Executive egos get involved and shit gets ugly. Again, complicated and will definitely vary from publisher to publisher.

Just one last thing, my experience in QA has been mostly positive. The hours can suck, the people can be even worse, but I'd never trade my 100+hr weeks for anything. The war stories are what makes things fun. I've gotten to work on a wide variety of products and it's really pushed my ability to think critically, solve problems, manage stress, build relationships with coworkers and a whole myriad of other useful life and job skills.

Not that anybody ever really cares about testers, but I like talking about this sort of stuff, so if people have questions, I'm more than willing to answer nonspecific stuff. ;)

Anyway, I'm going to shut up now for risk of just rewriting the article. :)

Edit: I lied about the shutting up thing. Another problem with testing is one of perception. A large segment of people (from Executives down to entry level designers) think they can test a game and that involves little more than "playing" for 8 hours a day. It's really unfortunate that the perception still exists, especially considering there is such a vast difference between play testers and QA testers. While play (balance, flow, progression) testing is part of the QA experience, the job extends to things far beyond them and the sad thing is that in my experience developers (Producers, Leads, Execs) are less likely to listen to game play feedback from QA testers than they are from focus groups. There are reasons behind that, I'm sure. But when somebody that is familiar with every in and out of your product says something is screwy with an aspect of "subjective" game play, I would think the decision makers and the product owners would listen...
 

Girlysprite

New member
Nov 9, 2007
290
0
0
I appreciate testers, greenlaser :) I am a designer in a game company, and I remember the months that the game went in full-testing all too well. It's surprising how many bugs a level can have. Testing a game can be a real bore, so i can appreciate how people manage to play a level for the hundredth time and still are sharp enough to find a new bug.

Testing is more then 'just play around a bit'. You need to know what you are looking for, and how to find it.
 
Feb 13, 2008
19,430
0
0
Thinking of a certain game, testers didn't even check if players would enhance all their powers with Damage Boosters. Having devious, leet-speaking, spawn-camping, gold farmers is the way to go.
 

PEWPEWGreenLaser

New member
Jul 23, 2008
45
0
0
The_root_of_all_evil said:
Thinking of a certain game, testers didn't even check if players would enhance all their powers with Damage Boosters. Having devious, leet-speaking, spawn-camping, gold farmers is the way to go.
I have to wonder; did you work on said game? If not, why assume that it was testers that missed something. 9 times out of 10, this isn't the case and the decision to fix or not fix is handed down from way above the testing level. For major things or even the minor little gripes people have with games, to fix or not to fix is just not something testers have control over. As a matter of fact, it's really not even as simple as pointing a finger at a single position in the development cycle and saying "omgz. ur fault!"
 
Feb 13, 2008
19,430
0
0
PEWPEWGreenLaser said:
I have to wonder; did you work on said game? If not, why assume that it was testers that missed something. 9 times out of 10, this isn't the case and the decision to fix or not fix is handed down from way above the testing level. For major things or even the minor little gripes people have with games, to fix or not to fix is just not something testers have control over. As a matter of fact, it's really not even as simple as pointing a finger at a single position in the development cycle and saying "omgz. ur fault!"
Well, I did Beta test it. I also mailed them the comment that 'Perhaps it might be a good idea to test it'. I watched as they had to bring in a massive patch halfway through Alpha to fix that very problem and I watched them blame the testers.

So...I'm putting my bets on 'quite sure' especially as the idea had been on the forums on Beta, and six months into Alpha.

It's not the only huge design flaw I spotted; and I'm really not looking that hard. If you have a game that requires you to do X to Y, and then you give players chance to boost X or boost V; you have to be thinking they'll want to boost X.
 

shMerker

New member
Oct 24, 2007
263
0
0
To those think that QA testers "play games all day" consider this analogy. You can QA test a piano by hitting every single key, listening to the note it makes, and writing down which notes are off and by how much(You also have to include detailed instructions on how to hit that key, preferably with attached photographs and a video of you hitting the key). If you think that that is "playing a piano" then you are right, QA testing is "playing games."
 

dgrassa

New member
Jul 24, 2008
1
0
0
Granted I only have one day of testing experience, it was a 14 hour day. It sounds to me like the experience at Nintendo and Microsoft are understandably ideal compared to other places. Working for a third party company, you can be expected to pull 50-70 hour weeks frequently, and with a 6 month contracted that won't be renewed unless you put in every hour that's asked of you. Not taking the overtime can be considered to be taking vacation time. You're in a dark room for 10-16 hour days with temperatures that are either freezing or boiling hot depending on when the cooling systems kick in, and replaying the same level of a game 15-20 times a day. As somebody who could happily play video games for entertainment every day until I die, I'd say doing QA/Testing is poison for being able to enjoy games. There's too much of a good thing, and then there's too much of a not-so-good, unpolished and stressful thing.
 

PEWPEWGreenLaser

New member
Jul 23, 2008
45
0
0
dgrassa said:
Granted I only have one day of testing experience, it was a 14 hour day. It sounds to me like the experience at Nintendo and Microsoft are understandably ideal compared to other places. Working for a third party company, you can be expected to pull 50-70 hour weeks frequently, and with a 6 month contracted that won't be renewed unless you put in every hour that's asked of you. Not taking the overtime can be considered to be taking vacation time. You're in a dark room for 10-16 hour days with temperatures that are either freezing or boiling hot depending on when the cooling systems kick in, and replaying the same level of a game 15-20 times a day. As somebody who could happily play video games for entertainment every day until I die, I'd say doing QA/Testing is poison for being able to enjoy games. There's too much of a good thing, and then there's too much of a not-so-good, unpolished and stressful thing.
Spot on. Only people that have really tested video games know what it's like. Microsoft and Nintendo may be the goal for all testing labs to shoot for, but the reality of the situation is that most places just aren't sunshine and rainbows all day.

As far as the too much of a good thing goes, you reach a saturation point with testing where you're able to compartmentalize "playing" for pay and playing for fun. It took me about a year of hellish work schedules (almost 6 months straight of 12hr/day overtime, with over 450hrs clocked one January...) to reach that point. I'm sort of a workaholic though.

The (at the time) Director of my department and my manager took one look at me on my 29th straight day of 16's and said "Go home. You look dead." I really did, but I was a contract employee and sort of terrified that it was the beginning of the end of my tenure with that company. There's also something a little disheartening about being given "special" treatment when the rest of your team is still in the shit, so to speak. It sucked terribly at the time, but it's honestly one of my favorite war stories. :)
 

shMerker

New member
Oct 24, 2007
263
0
0
I remember after a month or so of testing I started being ultra critical of the games I was playing on my own time. I also remember being at a party and having this conversation:

Stranger: So what do you guys do?
Me: I test video games.
Friend: I'm a paramedic.
Stranger(turning to me):That's cool. What's it like?
Me: He's the one that saves lives!
 

papamook

New member
Jul 25, 2008
3
0
0
its a nightmare testing a game that's not fun at all, especially when it doesn't get any more fun with following iterations!