72: "You're Wrong"
- Thread starter The Escapist Staff
- Start date
Not always are we wrong - I would say that Ubisoft's decision to hold back the release of Heroes V till they could polish it based on community feedback (at least according to them) was the right one. The game wasn't ready, and the community wasn't screaming about features they wanted so much as about it not being ready yet.
The hardcore may not always be wrong, but they (we) are always safely ignored.
The hardcore complain about everything; so we can't just point to one or two cases where their complaints were justified as some sort of proof that they (we) have any credibility.
What made holding back Heroes V the right decision, was the inclusion of moderate voices among the cacophony. Even ignoring the hardcore, Ubi would've still been hearing that Heroes needed more work.
The most important thing for developers to remember --the place where I disagree with this article altogether-- is that official message boards are a waste of time altogether.
They're an ineffective, error-prone, and expensive method of official communication. All the benefits of messageboards can be easily gleaned by reading the community boards. All the benefits of official communication can be had by actually publishing information. Official boards are a waste of time and resources.
Right: Mythic and the Herald.
Wrong: Blizzard and blue-posts.
The hardcore complain about everything; so we can't just point to one or two cases where their complaints were justified as some sort of proof that they (we) have any credibility.
What made holding back Heroes V the right decision, was the inclusion of moderate voices among the cacophony. Even ignoring the hardcore, Ubi would've still been hearing that Heroes needed more work.
The most important thing for developers to remember --the place where I disagree with this article altogether-- is that official message boards are a waste of time altogether.
They're an ineffective, error-prone, and expensive method of official communication. All the benefits of messageboards can be easily gleaned by reading the community boards. All the benefits of official communication can be had by actually publishing information. Official boards are a waste of time and resources.
Right: Mythic and the Herald.
Wrong: Blizzard and blue-posts.
I'm not a hardcore gamer, but I'm still very critical of game design as I used to develop and design software (I've written and maintained my fair share of design documents).
I do have to say that I respect the sentiment of this article. I want nothing more than for developers to be accountable for and protective of their game designs. For that, I can appreciate this article.
However, the article is written in such a way that it focuses on the most extreme (and seemingly amateurish) situation between beta-tester and designer/developer. Do game developers typically fall prey to the demands of beta-testers? I need some real examples of bad design choices influenced by beta-testers outside the scope of identifying bugs, performance issues, and challenge/balance concerns to appreciate the crux of this article.
I do have to say that I respect the sentiment of this article. I want nothing more than for developers to be accountable for and protective of their game designs. For that, I can appreciate this article.
However, the article is written in such a way that it focuses on the most extreme (and seemingly amateurish) situation between beta-tester and designer/developer. Do game developers typically fall prey to the demands of beta-testers? I need some real examples of bad design choices influenced by beta-testers outside the scope of identifying bugs, performance issues, and challenge/balance concerns to appreciate the crux of this article.
This may be the way in PC game land, but in every company I've worked for (ok, it's only two, but you don't need to know that) making console games have had focus groups.
We put an advert in numerous sources, some aimed at the 'hardcore' - signs in EB for example - and others not so hardcore - we ask our friends and relatives.
This is a much better process for giving feedback - we have casual gamers as well as hardcore gamers.
Anyway, more of the stupid decisions get made by execs hired by companies because they're (supposedly) good managers or marketeers, but who really have little knowledge of the games and the games industry, than by hardcore gamers.
We put an advert in numerous sources, some aimed at the 'hardcore' - signs in EB for example - and others not so hardcore - we ask our friends and relatives.
This is a much better process for giving feedback - we have casual gamers as well as hardcore gamers.
Anyway, more of the stupid decisions get made by execs hired by companies because they're (supposedly) good managers or marketeers, but who really have little knowledge of the games and the games industry, than by hardcore gamers.
The problem with message boards is that many people use them to express, and gain recognition, with regards to their thoughts on whatever the subject may be. If you type "I didn't like a weapon in this game" on a board your not going to get any recognition, where as if you say "They have totally destroyed this game by designing the crappiest pea shooter weapons ever seen in a video game - congradulations you all suck!" you may get more responses (and recognition) as its more emotive, but you're thoughts are no longer realistic. Therefore it would be ridiculous to use a message board for improvements - or as a place to provide feedback to you're beloved game developer. They are just a marketing tool.
How would a focus group which uses friends and relatives of people who made the game be of any use? I mean if you're in a focus group for a game your cousins been programming for ages, are you gonna tell him/her it's the biggest piece of crap you've ever seen?
How would a focus group which uses friends and relatives of people who made the game be of any use? I mean if you're in a focus group for a game your cousins been programming for ages, are you gonna tell him/her it's the biggest piece of crap you've ever seen?
Another instance of Ubisoft correctly listening to its community was with the nextgen version of the recently released Splinter Cell. Initially the old Spy Versus Merc mode was going to be replaced with a new Spy Versus Spy mode. Then a significant portion of the message boards adopted the signature, "Spy VS Spy Means I Don't Buy."
Ubisoft listened. Spy Vs Spy was relegated to the lastgen systems -- turning out to be just as silly as the boards imagined -- and a new Spy Vs Merc mode was made for the nextgen. The resulting Versus mode from Annecy was the only redeeming factor on the disc, offsetting Ubisoft Shanghai's inability to design a decent singleplayer.
Perhaps this is just an exception to the rule, or maybe your point doesn't apply when a significant portion of the community cries out about a single, specific issue.
Ubisoft listened. Spy Vs Spy was relegated to the lastgen systems -- turning out to be just as silly as the boards imagined -- and a new Spy Vs Merc mode was made for the nextgen. The resulting Versus mode from Annecy was the only redeeming factor on the disc, offsetting Ubisoft Shanghai's inability to design a decent singleplayer.
Perhaps this is just an exception to the rule, or maybe your point doesn't apply when a significant portion of the community cries out about a single, specific issue.
In my opinion if the game is a sequel then the developers should keep an open mind to what the fanbase is saying. Not only will the developers get a better idea of what the fans think needs to be fixed but they will get an idea of what made the original work in the first place.
In the case of Swat 3 none of the original developers worked on the sequel. Some of the developers admitted after Swat 4's release that they didn't really spend much time playing Swat 3 (if at all). They really didn't understand what made Swat 3 work as a game and what made it different from every generic first person shooter out there.
The result of not knowing the original product was a sequel that was watered down and lackluster.
In the case of Swat 3 none of the original developers worked on the sequel. Some of the developers admitted after Swat 4's release that they didn't really spend much time playing Swat 3 (if at all). They really didn't understand what made Swat 3 work as a game and what made it different from every generic first person shooter out there.
The result of not knowing the original product was a sequel that was watered down and lackluster.
I find that once the developers start listening to their fanbase, the quality of the products starts to go down. That's why when I find clues that the developers start catering to their fanbase, I usually stop paying attention and move on to something else.
The author makes a valid point about taking hardcore players with a grain of salt, but she makes it sound like game developers should make their games in a sealed room shut off from anyone outside the company until the game is content-complete. Patently untrue. The article should be careful to differentiate between public beta testing and playtesting input gained from outsiders who come into the office to play the game in an alpha state and give feedback to the developers.
Developers always need to be on that feedback loop, getting input from real people who haven't played the game before. Indeed, those testers should be as representative of your target market as possible, but developing for a mass market cannot be done in a vacuum. To use an example, if you think Valve developed Half Life 2 and Portal in complete seclusion before finishing and saying "voila!" like some master sculpter, you would be dead wrong. Valve carefully tunes and even redesigns both gameplay and content in response to input they get from their alpha testers. It is up to the developer to interpret feedback however they wish, but Portal would have been an utterly different and very likely much worse game had Valve not put it in the hands of people who's view of the product was not clouded by the responsibility for its creation.
Developers always need to be on that feedback loop, getting input from real people who haven't played the game before. Indeed, those testers should be as representative of your target market as possible, but developing for a mass market cannot be done in a vacuum. To use an example, if you think Valve developed Half Life 2 and Portal in complete seclusion before finishing and saying "voila!" like some master sculpter, you would be dead wrong. Valve carefully tunes and even redesigns both gameplay and content in response to input they get from their alpha testers. It is up to the developer to interpret feedback however they wish, but Portal would have been an utterly different and very likely much worse game had Valve not put it in the hands of people who's view of the product was not clouded by the responsibility for its creation.
Leaving aside the whole bit about female gamers (who may avoid places like GameFAQs, which are a hive or scum and villainy, but have no qualms about posting on the official forums of a game they play and enjoy), I would say the biggest problem with this article is that the thesis is "You need to ignore the hardcore players. Hardcore players are the ones who post on forums. Therefore, ignore anything said on your forums".
It doesn't matter how many posts there are on an issue, what tone they're presented in, or if they're backed up by statistics and FRAPS recordings. It's on the forums, so it's stuff you shouldn't listen to. You can make a box pop-up in-game during the beta test for people to write feedback in, but feel free to ignore what people write in there, too.
This basically kills any dialogue between the developers and the people using their product.
So what does the author propose to use instead? Focus groups.
Steve Krug has this to say about focus groups:
When the last-minute request is for a focus group, it's usually a sign that the request originated in Marketing. (...) As the launch date approaches, the Marketing people may feel that their only hope of sanity prevailing is to appeal to a higher authority: research. And the kind of research they know is focus groups.
(...) Focus groups can be great for determining what your audience wants, needs and likes - in the abstract. They're good for testing whether the idea behind the site makes sense and your value proposition is attractive. (...)
But they're not good for learning about whether your site works and how to improve it.
Now, granted, Steve Krug is a specialist on usability as applied to websites. But his idea of testing a website - take someone who's never seen it, sit them down and watch them try to muddle through - can very easily be applied to MMOs. You want to know if many thousands of people will be able to have fun playing your game? Get several thousand people and ask them to try and play your game.
Yes, there will always be the hardcore players, the malcontents, and the unpleasables. But not everyone on your forum is one of those. Some of them have valid points. Some of them have great ideas that you only *wish* you could come up with. Some of them have way too much time on their hands and will make you long, detailed reports with graphs and statistical analysis, for free.
Dismiss that sort of dedication to your game at your own peril.
It doesn't matter how many posts there are on an issue, what tone they're presented in, or if they're backed up by statistics and FRAPS recordings. It's on the forums, so it's stuff you shouldn't listen to. You can make a box pop-up in-game during the beta test for people to write feedback in, but feel free to ignore what people write in there, too.
This basically kills any dialogue between the developers and the people using their product.
So what does the author propose to use instead? Focus groups.
Steve Krug has this to say about focus groups:
When the last-minute request is for a focus group, it's usually a sign that the request originated in Marketing. (...) As the launch date approaches, the Marketing people may feel that their only hope of sanity prevailing is to appeal to a higher authority: research. And the kind of research they know is focus groups.
(...) Focus groups can be great for determining what your audience wants, needs and likes - in the abstract. They're good for testing whether the idea behind the site makes sense and your value proposition is attractive. (...)
But they're not good for learning about whether your site works and how to improve it.
Now, granted, Steve Krug is a specialist on usability as applied to websites. But his idea of testing a website - take someone who's never seen it, sit them down and watch them try to muddle through - can very easily be applied to MMOs. You want to know if many thousands of people will be able to have fun playing your game? Get several thousand people and ask them to try and play your game.
Yes, there will always be the hardcore players, the malcontents, and the unpleasables. But not everyone on your forum is one of those. Some of them have valid points. Some of them have great ideas that you only *wish* you could come up with. Some of them have way too much time on their hands and will make you long, detailed reports with graphs and statistical analysis, for free.
Dismiss that sort of dedication to your game at your own peril.
Meagan, I think you're mistaking the intent of the article. It's not saying that studios shouldn't test their games with unbiased parties, it's saying that forum feedback is largely useless because a) it's not a representative sample of the fans, it's self-selecting for the loudest and most boisterous, b) many of the opinions expressed are "heat of the moment" ones and not considered, dispassionate ones, c) too often complaints about a product in development posted by hardcore players don't take into consideration that the product is in development, and that distorts the feedback provided, and d) hardcore players aren't unbiased parties, frequently having their own prejudices such as exclusion of "noobs" or a desire to revisit a previous title.
Game forums are lousy places to do product research.
The right place to do product research is either some sort of controlled environment where biases can be accounted for, or "the wild" by monitoring actual gameplay. (And NOT gameplay as related in forums; that'd be like taking a fish census by asking fishermen about the ones that got away, alas.) This article [http://blogs.smh.com.au/screenplay/archives/018731.html] discusses the user test regimes for the Halo series, as does this article from Wired [http://www.wired.com/gaming/virtualworlds/magazine/15-09/ff_halo]... I point to these because I have them to hand, though I'd love to see how that regime compares to those of Valve (especially their monitoring of actual player experiences through Steam), BioWare, Blizzard, and other AAA studios.
I've been neck-deep in game forums for years now, and I beg any burgeoning game studio to disregard advice gained through these forums... it's not all poisonous, but unlike mushrooms it's too damned hard to tell the difference between the toxic and the beneficial strains.
-- Steve
Game forums are lousy places to do product research.
The right place to do product research is either some sort of controlled environment where biases can be accounted for, or "the wild" by monitoring actual gameplay. (And NOT gameplay as related in forums; that'd be like taking a fish census by asking fishermen about the ones that got away, alas.) This article [http://blogs.smh.com.au/screenplay/archives/018731.html] discusses the user test regimes for the Halo series, as does this article from Wired [http://www.wired.com/gaming/virtualworlds/magazine/15-09/ff_halo]... I point to these because I have them to hand, though I'd love to see how that regime compares to those of Valve (especially their monitoring of actual player experiences through Steam), BioWare, Blizzard, and other AAA studios.
I've been neck-deep in game forums for years now, and I beg any burgeoning game studio to disregard advice gained through these forums... it's not all poisonous, but unlike mushrooms it's too damned hard to tell the difference between the toxic and the beneficial strains.
-- Steve