I was going to comment on the article, but first...
There's at least three sets of testers for most games -- Developer, Publisher, and in the case of console games, Certification. This sounds mostly like Publisher testing, which is primarily black box for very political reasons (mainly because over-worked developers can snap when some little tester they don't know starts telling them how to do their job, whether they're right or not).
Developer-end testing is primarily white-box, and their job is initially to simply test the code (particularly fixes) just after they get put in. Once full-testing gets up, they end up primarily reproducing bugs found by black-box testers so the developer in question get get additional information about it, or see the issue in action (please note: developers, on average, do not play their own game often, and some never do, which is something that has eternally baffled me). In some cases, these guys will also fix quick typo and scripting bugs, but often they aren't allowed to (more political reasons, generally, though also some practical ones of keeping the number of people with direct access to the code down, and to ensure those changes are put in by someone who will know if they might have side-effects).
Publisher-end testing is almost always entirely black-box -- and in some cases, the tester is specifically ordered to play dumb (e.g. "even if you think this issue is related to another, don't act like it, write both up separately, and don't even refer to the other one in your write-up"). Some of this is directly practical (if the issues aren't actually related, confusion ensues if they get written up together), but mostly it's maintaining the black-box mind-set of not assuming anything about the underlying structure of the code.
Certification Testing, from what I know of it, is almost entirely about checking for very technical aspects of the game -- things like ensuring common terminology (e.g. calling it the "Left Analog Stick" and not "Left Thumb Stick" in the controller options menu), button prompts (nothing annoys a console maker more than seeing the buttons that look like those on the rival console), and trademarked words are used properly (e.g. PlayStation® always has a capital P and S, and if in an all-capitals font must still have a larger P and S than the other letters), and so on. They also do a quick shakedown to ensure the game doesn't crash regularly or have any really glaring functionality problems (they don't care if it's fun, just if it's playable).
And as a final comment to root_of_all_evil -- as the low man on the totem pole, testers are almost universally blamed even when they can directly point to the issue they wrote up on something. Alternately, the testers did test it and since it fit the design document they were given ("Boosters can be applied to powers? Check. All powers can be boosted? Check. Any limits on boosting the the official design? Nope? Check. Ok, next test suite."), didn't write it up because it was by design (despite any problems it may have seemingly caused), and were told not to question the almighty game and balance designers who clearly knew more, even if they never played their own game.
I have never seen a game shipped that didn't have some known issues. I have also never seen a game ship without several bugs entered regarding play balance, exploits, and design flaws which were waived by the developers and producers as either by-design / intentional, marginal / not worth the time to fix, or "too difficult to do" which end up being in every review as a bad mark, and in the case of multiplayer games wreak all sorts of havoc. I have never seen a review note something we missed, and I can't think of any cases where a major exploit showed up that we didn't catch first. My bet is on there being someone between the devs and the testers that caused the issue you described in said game, and not it being simply missed by the testers entirely unless it only had very limited in-house pre-alpha testing.
What? Have you never heard of Microsoft Flight Simulator? Mech Commander? Allegiance? Age of Empires? Halo? Those are all made by studios wholly owned by Microsoft (or which were wholly owned at the time of release). Microsoft Games Studio owns several studios, and publishes for many more. Which leads into my comments on the article, which focuses on one small part of testing...Eagle Est1986 said:
There's at least three sets of testers for most games -- Developer, Publisher, and in the case of console games, Certification. This sounds mostly like Publisher testing, which is primarily black box for very political reasons (mainly because over-worked developers can snap when some little tester they don't know starts telling them how to do their job, whether they're right or not).
Developer-end testing is primarily white-box, and their job is initially to simply test the code (particularly fixes) just after they get put in. Once full-testing gets up, they end up primarily reproducing bugs found by black-box testers so the developer in question get get additional information about it, or see the issue in action (please note: developers, on average, do not play their own game often, and some never do, which is something that has eternally baffled me). In some cases, these guys will also fix quick typo and scripting bugs, but often they aren't allowed to (more political reasons, generally, though also some practical ones of keeping the number of people with direct access to the code down, and to ensure those changes are put in by someone who will know if they might have side-effects).
Publisher-end testing is almost always entirely black-box -- and in some cases, the tester is specifically ordered to play dumb (e.g. "even if you think this issue is related to another, don't act like it, write both up separately, and don't even refer to the other one in your write-up"). Some of this is directly practical (if the issues aren't actually related, confusion ensues if they get written up together), but mostly it's maintaining the black-box mind-set of not assuming anything about the underlying structure of the code.
Certification Testing, from what I know of it, is almost entirely about checking for very technical aspects of the game -- things like ensuring common terminology (e.g. calling it the "Left Analog Stick" and not "Left Thumb Stick" in the controller options menu), button prompts (nothing annoys a console maker more than seeing the buttons that look like those on the rival console), and trademarked words are used properly (e.g. PlayStation® always has a capital P and S, and if in an all-capitals font must still have a larger P and S than the other letters), and so on. They also do a quick shakedown to ensure the game doesn't crash regularly or have any really glaring functionality problems (they don't care if it's fun, just if it's playable).
And as a final comment to root_of_all_evil -- as the low man on the totem pole, testers are almost universally blamed even when they can directly point to the issue they wrote up on something. Alternately, the testers did test it and since it fit the design document they were given ("Boosters can be applied to powers? Check. All powers can be boosted? Check. Any limits on boosting the the official design? Nope? Check. Ok, next test suite."), didn't write it up because it was by design (despite any problems it may have seemingly caused), and were told not to question the almighty game and balance designers who clearly knew more, even if they never played their own game.
I have never seen a game shipped that didn't have some known issues. I have also never seen a game ship without several bugs entered regarding play balance, exploits, and design flaws which were waived by the developers and producers as either by-design / intentional, marginal / not worth the time to fix, or "too difficult to do" which end up being in every review as a bad mark, and in the case of multiplayer games wreak all sorts of havoc. I have never seen a review note something we missed, and I can't think of any cases where a major exploit showed up that we didn't catch first. My bet is on there being someone between the devs and the testers that caused the issue you described in said game, and not it being simply missed by the testers entirely unless it only had very limited in-house pre-alpha testing.
Not that there is anything wrong with that haha. In any case I always imagined guys sitting around eating pizza and playing games, turning every now and then to give a big thumbs up or thumbs down to their own reflection in a one way mirror with a bunch of scientist types taking notes. You certainly opened my eyes hehe.