Should all games need to have mandatory QA testing before release?

[CritialGaming]

Let me give you guys an example of how AAA game business works.

1. The Developer comes to a publisher with a game idea.
2. The Publisher approves the game idea.
3. The publish then gives the developer a budget and a time frame.
4. The developer starts making the game.
5. Several months, even a year or more into the process depending on the scope of the game and time frame, the developers have a playable alpha ready for testing.
6. Developer and Publishers both have QA departments and it is at this point that they really start testing the game.
7. Next the game goes into BETA once big issues are fixed and the game is feature complete.
8. At this point in Development, the time deadline is getting close.
9. As QA finds issues, Developers fix the issues and issues new builds of the game. This typically happens about 10-20 times, each time the developer has to start deciding on what issues to fix and what issues to ignore. They do this both to make sure that they meet deadline, but also to screw with the game code as little as possible. Every issue that is fixed, has a risk of causing ten other issues.

Ultimately the decision to fix bugs boils down to three things.

1. How long will the issue take to fix.
2. How likely is the average player going to encounter this issue.
3. If the player finds the issue, does it break the game?

That's the scoop on why we get buggy games. And really the only way to fix the issue, is to remove deadlines (within) reason from Developers so that proper time can be dedicated to QA and bug fixing.

 

[Fappy]

Topics like this are always full of hilarious posts from people who have no concept of how software development actually works. At least a few people know what's up more or less.

I remember the Game Grumps talking about how Sonic Boom's QA people actually told them they pointed out many of these flaws but were ignored. No sympathy for them though. QA means Quality -Assurance-. If you don't ASSURE they listen to you, you failed your job.

QA people aren't typically the same people that actually fix the bugs. In most companies that I know of the QA department runs tests and passes the results to R&D. It's R&D's job to act on their feedback. If QA's feedback is not addressed it is almost always the fault of either R&D or the department managers (or higher ups pushing deadlines/not hiring enough staff). It could be some large companies have more fragmented departments, but this principle is generally consistent across the software industry.

I think if a bug is found and it doesn't get fixed in a certain amount of time (let's say 2 weeks) the people responsible for releasing a broken product should have to pay a hefty fine, because losing money will teach them, not the whining of the costumers who often don't learn a thing and buy the next big bug fest day 1, just to whine again.

Who would have the authority to fine them? If enough people backed it you could file a class action lawsuit, of course. That's the only legal way to pursue something like this.

EDIT: It is my perception atm that QA uses a letter grading system for classifying bugs, I might be wrong though.

I think every company does it differently. Where I work we just use 'Minor', 'Major', and 'Critical' to specify bug urgency/impact.

Nah, people should just wise up and not buy games on release.

People shouldn't be afraid to buy games on release day, but they should definitely be mindful about the risks. If the company has a history of poor launches... don't buy their game at launch. And if you do, don't come here bitching about it every time you make the same mistake.

 

[Saelune]

Topics like this are always full of hilarious posts from people who have no concept of how software development actually works. At least a few people know what's up more or less.

I remember the Game Grumps talking about how Sonic Boom's QA people actually told them they pointed out many of these flaws but were ignored. No sympathy for them though. QA means Quality -Assurance-. If you don't ASSURE they listen to you, you failed your job.

QA people aren't typically the same people that actually fix the bugs. In most companies that I know of the QA department runs tests and passes the results to R&D. It's R&D's job to act on their feedback. If QA's feedback is not addressed it is almost always the fault of either R&D or the department managers (or higher ups pushing deadlines/not hiring enough staff). It could be some large companies have more fragmented departments, but this principle is generally consistent across the software industry.

I never said or meant to suggest that they fix them. I'm just saying, the word Assure is the A in QA. Technically this means QA should also be watching over the bug fixers with whips until its all fixed. In reality though, they need a different title. Either way its a group effort, good or bad.

 

[Fappy]

Topics like this are always full of hilarious posts from people who have no concept of how software development actually works. At least a few people know what's up more or less.

I remember the Game Grumps talking about how Sonic Boom's QA people actually told them they pointed out many of these flaws but were ignored. No sympathy for them though. QA means Quality -Assurance-. If you don't ASSURE they listen to you, you failed your job.

QA people aren't typically the same people that actually fix the bugs. In most companies that I know of the QA department runs tests and passes the results to R&D. It's R&D's job to act on their feedback. If QA's feedback is not addressed it is almost always the fault of either R&D or the department managers (or higher ups pushing deadlines/not hiring enough staff). It could be some large companies have more fragmented departments, but this principle is generally consistent across the software industry.

I never said or meant to suggest that they fix them. I'm just saying, the word Assure is the A in QA. Technically this means QA should also be watching over the bug fixers with whips until its all fixed. In reality though, they need a different title. Either way its a group effort, good or bad.

If the QA folks have time, sure. It's a workflow issue though. They spend all of their time testing and writing bug reports. The amount of issues they have to process for a given software release can be anywhere between "staggering" and "monumental" in most cases I have personally witnessed. The people with the whips should be the department managers or other departments charged with overseeing delivery of the product (such as a product manager).

 

[CritialGaming]

Topics like this are always full of hilarious posts from people who have no concept of how software development actually works. At least a few people know what's up more or less.

I remember the Game Grumps talking about how Sonic Boom's QA people actually told them they pointed out many of these flaws but were ignored. No sympathy for them though. QA means Quality -Assurance-. If you don't ASSURE they listen to you, you failed your job.

QA people aren't typically the same people that actually fix the bugs. In most companies that I know of the QA department runs tests and passes the results to R&D. It's R&D's job to act on their feedback. If QA's feedback is not addressed it is almost always the fault of either R&D or the department managers (or higher ups pushing deadlines/not hiring enough staff). It could be some large companies have more fragmented departments, but this principle is generally consistent across the software industry.

I never said or meant to suggest that they fix them. I'm just saying, the word Assure is the A in QA. Technically this means QA should also be watching over the bug fixers with whips until its all fixed. In reality though, they need a different title. Either way its a group effort, good or bad.

Yeah I don't think you realize how low on the totem pole QA people are. The individual tester has no say as to whether their bug is fixed or not. On a typical project you have only a Lead Tester, and that is the only guy who has a communication line with the dev open. Most of the time the devs decide if a bug gets fixed. However the Lead Tester can push the severity of bug or two here and there to make them fix those bugs.

Most of the devs just ignore QA as much as they possibly can, fixing only enough to get through certification. Hell sometimes they don't even do that.

I was working on COD Modern Warfare for XBOX 360, and that game ignored EVERY rule Microsoft gave us for releasing on Xbox. It had profile swapping issues, it had labeling issues, it had bad button prompts, the fucking multiplayer did nothing right. And we told them do not submit this to MS, because it would get rejected until this huge list of issues were fixed.

They ignored us and submitted the game anyway....and they passed.

The story spread, and the next game we told not to submit because of Xbox Restrictions. I forgot what game it actually was, but they ignore us, and Xbox failed the game, forcing the devs to go back and fix our issues before release. It ate 45,000 dollars out of the budget for no reason.

We learned quickly that if a game is fun, or really fucking awesome looking (like COD), First party certification teams would let certain things go so that the could get the game on their console. Microsoft ended up ignoring their own rules and letting Modern Warfare pass, because Playstation already passed the game, and if they had rejected the game, then Xbox would NOT have gotten Modern Warfare until six weeks AFTER Playstation got it.

Politics.

 

[veloper]

Let the market take care of it.

If gamers continue to buy broken products, then we simply deserve this shit, but if gamers start avoiding the worst offenders and work up from there, then game quality will improve.

 

[Fappy]

Topics like this are always full of hilarious posts from people who have no concept of how software development actually works. At least a few people know what's up more or less.

I remember the Game Grumps talking about how Sonic Boom's QA people actually told them they pointed out many of these flaws but were ignored. No sympathy for them though. QA means Quality -Assurance-. If you don't ASSURE they listen to you, you failed your job.

QA people aren't typically the same people that actually fix the bugs. In most companies that I know of the QA department runs tests and passes the results to R&D. It's R&D's job to act on their feedback. If QA's feedback is not addressed it is almost always the fault of either R&D or the department managers (or higher ups pushing deadlines/not hiring enough staff). It could be some large companies have more fragmented departments, but this principle is generally consistent across the software industry.

I never said or meant to suggest that they fix them. I'm just saying, the word Assure is the A in QA. Technically this means QA should also be watching over the bug fixers with whips until its all fixed. In reality though, they need a different title. Either way its a group effort, good or bad.

Yeah I don't think you realize how low on the totem pole QA people are. The individual tester has no say as to whether their bug is fixed or not. On a typical project you have only a Lead Tester, and that is the only guy who has a communication line with the dev open. Most of the time the devs decide if a bug gets fixed. However the Lead Tester can push the severity of bug or two here and there to make them fix those bugs.

Most of the devs just ignore QA as much as they possibly can, fixing only enough to get through certification. Hell sometimes they don't even do that.

I was working on COD Modern Warfare for XBOX 360, and that game ignored EVERY rule Microsoft gave us for releasing on Xbox. It had profile swapping issues, it had labeling issues, it had bad button prompts, the fucking multiplayer did nothing right. And we told them do not submit this to MS, because it would get rejected until this huge list of issues were fixed.

They ignored us and submitted the game anyway....and they passed.

The story spread, and the next game we told not to submit because of Xbox Restrictions. I forgot what game it actually was, but they ignore us, and Xbox failed the game, forcing the devs to go back and fix our issues before release. It ate 45,000 dollars out of the budget for no reason.

We learned quickly that if a game is fun, or really fucking awesome looking (like COD), First party certification teams would let certain things go so that the could get the game on their console. Microsoft ended up ignoring their own rules and letting Modern Warfare pass, because Playstation already passed the game, and if they had rejected the game, then Xbox would NOT have gotten Modern Warfare until six weeks AFTER Playstation got it.

Politics.

Damn, that sucks. QA folks in enterprise software don't get treated that badly from what I have seen. The game industry just seems to chew people up and spit them out

 

[Saelune]

I never thought QA people were high up...in fact Id assume testers would do most of the bug finding, and they are the bottom tier since like, anyone can be a tester, despite what Sony wants you to think.

 

[Nielas]

Damn, that sucks. QA folks in enterprise software don't get treated that badly from what I have seen. The game industry just seems to chew people up and spit them out

Enterprise software is software that actually matters. If your software has bugs then your clients will not buy it and might even sue you if you cost them money. Games do not matter and "gamers" will buy all kinds of buggy software if it is marketed just right.

 

[Fappy]

Damn, that sucks. QA folks in enterprise software don't get treated that badly from what I have seen. The game industry just seems to chew people up and spit them out

Enterprise software is software that actually matters. If your software has bugs then your clients will not buy it and might even sue you if you cost them money. Games do not matter and "gamers" will buy all kinds of buggy software if it is marketed just right.

Yeah, enterprise software companies have a lot more incentive to deliver what was sold. Depending on the size of the company they also have the flexibility to support specific clients through services departments and/or partners. If an issue pops up for a customer, no problem, get your guys responsible for that customer on it and fix whatever is broke. Missing functionality that sales insisted exists, but actually doesn't? Scramble at the last second to make sure it's in the next software release!

Game devs don't really have this kind of flexibility

 

[Dreph]

As something mildly close to a developer I figure I should put my two cents in.

First a quick bit of background. I work on small scale tools, all internal. Our team has three "developers" and we all work on separate projects, so we are basically build products on our own.

Our QA team is awesome and finds pretty much everything and goes into a level of detail that is absolutely insane. Even on small scale tools we get back hugely detailed lists of problems and fixes we need to make. Guess what we do with that list. Triage it. I hate that. I should take that report and fix everything end to end, but I don't get to. Typically my manager meets with the stakeholder figures out what is make or break and we release it in whatever state they are ok with. It sucks.

There is far too much politics and money riding on release dates, even on small scale tools.

Anyway... Thought I should mention how it looks from my end and honestly I believe is sucks from pretty much all angles.

 

[Musou Tensei]

I think if a bug is found and it doesn't get fixed in a certain amount of time (let's say 2 weeks) the people responsible for releasing a broken product should have to pay a hefty fine, because losing money will teach them, not the whining of the costumers who often don't learn a thing and buy the next big bug fest day 1, just to whine again.

That is a horrible idea.

It is literally impossible for any software to be completely bug free, and trying to fix those bugs could even introduce more, possibly worse bugs, which is a problem made even worse with the introduction of a time limit with a huge penalty for failure.

Furthermore, how would you even define what a bug is in the context of this hypothetical policy? Is it only game breaking bugs that literally make it impossible to progress? What about occasional crashes? Harmless cosmetic bugs?

What about things like Missingno in Pokemon or all of the weird bugs and issues in Ocarina of time that speedrunners exploit to get those super short times? Are those okay, or would Nintendo have to pay a massive fine if they wanted to rerelease those games unchanged today?

Severe bugs that harm the enjoyability of a game. Also the price of the fine should be measured on the company's net worth, it should hurt then enough to teach them a lesson but not ruin them, the 1st time at least.

Let the market take care of it.

If gamers continue to buy broken products, then we simply deserve this shit, but if gamers start avoiding the worst offenders and work up from there, then game quality will improve.

It's not my fault that the mainstream is garbage that eats up everything that is popular, I tell them again and again to not do it but they won't listen.

 

[Saltyk]

I get where you're coming from. However, you have to realize it is not that easy. Some bugs can be obscure. The type that only occurs under insanely specific situations. Some of these are minor and ship as a "feature" or cheat.

For example, on PS1 there was the game Wild Arms. Now, once you had all three party member meet up, you could do a cheat for 255 of any item. All you had to do was have the first character use a Heal Berry. The second would do the same. The third, would switch the Heal Berry with whatever item you wanted 255 of and then defend. This basically happened because the game would "use" the item you switch. It would count two uses of it. If you only had one item, it would count from 1 to 0, but on the second use it would have to go below 0. But there is no number below 0, so instead it went to the highest number of items you could have. 255. Effectively 0-255 was a loop in the game's logic in this scenario. Logically, no one would do that under normal circumstances, but this glitch was published in gaming magazines and basically sold as a feature.

You see part of the problem is that changing the code can cause other problems. We like to think that game programming is as simple as plugging in some data and everything is fine. But games are complex. You may fix a minor issue and create a major one instead. FF7 is actually famous for having been so unstable that the moment they finally got the game stable, they launched it as it was. Didn't even take out the developer tools like normal.

 

[Ambient_Malice]

Nintendo had mandatory quality assurance testing during the N64 era. This applied to both game stability and visual quality. My memory is foggy in regards to the specifics, but I think titles underwent two days of extensive testing, and if a single crash or gamebreaking bug were encountered, the game would be bounced.

In addition, Nintendo had strict standards when it came to basic image quality. The N64 z-buffer could be disabled for a big performance boost, and it was for a handful of games, but Nintendo had a strict policy against z-fighting. Basically, if Skyrim were an N64 game, it wouldn't make it past 15 minutes of testing.

World Driver Championship was one game that disabled the z-buffer, and it had to be carefully designed to avoid z-fighting. The developers were afraid Nintendo would bounce the game because if you look closely, flaws can be seen, but it managed to get certified.


Indiana Jones and the Infernal Machine was released via Blockbuster in the USA, and the PAL version was cancelled. By sheer coincidence, the game was a rush job with a team that was gutted by the Battle for Naboo team leaving, allegedly, one developer working on it. Great game and great port that pushed the N64 very hard while keeping a stable-ish framerate and running at 400x440. But it crashes semi-randomly, and it is reasonably common to run into gamebreaking bugs. It is very, very likely that the reason the PAL version was cancelled (leaked a few years ago when someone was selling items from the THQ liquidation) was because the game was never, ever, ever, ever, ever going to make it past Nintendo's quality control testing.


Even the crappiest commercially released N64 games tended to meet basic standards such as not crashing or significant graphical issues. Nintendo had an odd approach where they were openly hostile towards third party developers by creating a difficult to work with console with poor documentation, while also demanding that developers create games that met their standards.

 

[SpiralLegacy]

I think if a bug is found and it doesn't get fixed in a certain amount of time (let's say 2 weeks) the people responsible for releasing a broken product should have to pay a hefty fine, because losing money will teach them, not the whining of the costumers who often don't learn a thing and buy the next big bug fest day 1, just to whine again.

That is a horrible idea.

It is literally impossible for any software to be completely bug free, and trying to fix those bugs could even introduce more, possibly worse bugs, which is a problem made even worse with the introduction of a time limit with a huge penalty for failure.

Furthermore, how would you even define what a bug is in the context of this hypothetical policy? Is it only game breaking bugs that literally make it impossible to progress? What about occasional crashes? Harmless cosmetic bugs?

What about things like Missingno in Pokemon or all of the weird bugs and issues in Ocarina of time that speedrunners exploit to get those super short times? Are those okay, or would Nintendo have to pay a massive fine if they wanted to rerelease those games unchanged today?

Severe bugs that harm the enjoyability of a game. Also the price of the fine should be measured on the company's net worth, it should hurt then enough to teach them a lesson but not ruin them, the 1st time at least.
.

That's an entirely subjective metric that does nothing to answer my question. How do you determine what bugs fit that criteria or not?

If a bug is not consistently game breaking, they will ship with it. It is on a relative stability scale that they determine whether or not they release with it. And a majority of bugs are corner cases that simply cannot be caught during the quality process. But small release windows exacerbate the possibility of more bugs being missed.

 

[WeepingAngels]

How about people stop buying buggy games. That means don't pre-order and wait until the reviews come out and people need to demand that reviewers mention bugs and only trust reviewers (professional or not) who do mention bugs consistently.

Pre-order are partly to blame for this, IMO.

 

[Fappy]

... and people need to demand that reviewers mention bugs and only trust reviewers (professional or not) who do mention bugs consistently.

This is actually a huge issue with the advent of game content youtubers. Since coverage of new games is shifting more towards popular youtube channels, a lot of publishers have been paying for sponsored content disguised as the channel's genuine content. In the case of Shadow of Mordor for example, youtubers were only given authorization to play and upload videos of the game before release so long as they followed a long list of stipulations, which included the omission of any and all bugs. That's right, if they encountered a bug it had to be edited out of the video and they couldn't mention it.

That's shady fucking shit and grounds for false advertising, imo.

 

[BeerTent]

Let the market take care of it.

If gamers continue to buy broken products, then we simply deserve this shit, but if gamers start avoiding the worst offenders and work up from there, then game quality will improve.

It's not my fault that the mainstream is garbage that eats up everything that is popular, I tell them again and again to not do it but they won't listen.

... and people need to demand that reviewers mention bugs and only trust reviewers (professional or not) who do mention bugs consistently.

This is actually a huge issue with the advent of game content youtubers. Since coverage of new games is shifting more towards popular youtube channels, a lot of publishers have been paying for sponsored content disguised as the channel's genuine content. In the case of Shadow of Mordor for example, youtubers were only given authorization to play and upload videos of the game before release so long as they followed a long list of stipulations, which included the omission of any and all bugs. That's right, if they encountered a bug it had to be edited out of the video and they couldn't mention it.

That's shady fucking shit and grounds for false advertising, imo.

I think this is the problem here...

As consumers, we've all turned around and said, "Hey... This is okay." And now we have the AAA industry. Nobody's holding anybody accountable. Spineless consumers are to blame.