Metacritic Co-Founder Calls on Reviewers to Rate More Bad Games

[Treblaine]

70% average, 50% fail? If it works for schools, it works for video games.

But it does beg the question, why bother making distinctions under 50%?

Games are more than just a large monetary investment, they are also a large investment of time, effort and frustration. You can watch a really bad movie with comparative ease as it is quite a passive and relatively short experience, if it's bad you can "disconnect" to a large extent.

If you "disconnect" from a shit game it becomes unplayable and hence un-review-able.

I think the most important "borderline" game for the 50/100 metascore for video games is Duke Nukem Forever, with the various platform versions hovering around 50/100. It's a well known and well played game that is generally poorly received, I think this would be the best game to explore that lower end of the spectrum.

Captcha: Queen's ediclu

 

[anonymity88]

I feel this picture is appropriate:

Spoiler

And I agree. I'm tired of seeing crap get passing scores, and I'm tired of great games being considered crap because their score isn't an 80 or higher.

That makes me even more upset because ilomilo is a pretty fun game, and sadly people will ignore it just because it didn't get in the 90 range.

The price argument comes into this as well. New DVD's tend to cost about £12 - £15, a full price game costs £40.

When you're taking the plunge with that much money you're going to find yourself suddenly becoming more discerning and less likely to take the plunge on a franchise you're unfamiliar with. Also explains the brand loyalty that has seen games like CoD become yearly cash cows. i.e. the safe bet.

 

[BlindChance]

I continue to argue that Clint Hocking nailed it when he called for a "5 Stars and the Truth" [http://clicknothing.typepad.com/Design/hockingc_GDC09_Microtalk_FiveStars.zip](Warning: File download with a powerpoint and word document) rating system. Such a system would be much harder for publications to 'game' and therefore promote more honesty.

Happily, the Escapist uses 5 Stars and the Truth!

 

[Jumplion]

Metacritic needs to die a fiery death. Anyone dumb enough to rely on metacritic should be in that fire.

Adam Sessler annoys me, but this video sums up nicely : http://www.youtube.com/watch?v=0QsXrswJ-yM

Don't blame the messenger for what is the fault of the perpetrators.

I don't really understand the hate towards Metacritic (I am shocked, but for once Jim makes some sense [http://www.escapistmagazine.com/videos/view/jimquisition/3607-Metacritic-Isnt-the-Problem]), and other review gathering sites, help get the general idea and feel towards a game. And even the video blames the publishers more, not Metacritic.

Honestly, this whole ratings system needs to be skewed back towards the middle. Crap games are given way too forgiving ratings and good games are condemned because they are .5 of a point off of a "9.0". Part of the problem, I think, is that games are a pretty expensive hobby so we are much more careful with our money when dealing with newer, fresher kind of games. I much prefer the 5 point scale if numbers must be used.

 

[koroem]

Metacritic needs to die a fiery death. Anyone dumb enough to rely on metacritic should be in that fire.

Adam Sessler annoys me, but this video sums up nicely : http://www.youtube.com/watch?v=0QsXrswJ-yM

Don't blame the messenger for what is the fault of the perpetrators.

I don't really understand the hate towards Metacritic (I am shocked, but for once Jim makes some sense [http://www.escapistmagazine.com/videos/view/jimquisition/3607-Metacritic-Isnt-the-Problem]), and other review gathering sites, help get the general idea and feel towards a game. And even the video blames the publishers more, not Metacritic.

Honestly, this whole ratings system needs to be skewed back towards the middle. Crap games are given way too forgiving ratings and good games are condemned because they are .5 of a point off of a "9.0". Part of the problem, I think, is that games are a pretty expensive hobby so we are much more careful with our money when dealing with newer, fresher kind of games. I much prefer the 5 point scale if numbers must be used.

I agree with Jim in that Metacritic itself has done no wrong, but it is based on a broken system and it's existence has ultimately done more harm then good in this industry. Whether it is the publishers to blame for interpreting the numbers wrong, or the reviewers for using them wrong, I still stand by my opinion.

Numbers should not be used. No way, no how. A scoring system for opinion reviewing is the most broken system possible and I cannot understand why it is used. A reviewer could slam a game to hell and back and give it a 60, but then another reviewer can love a different game and have to score it low due to technical problems and score it a 65.

What the hell is the difference? What does 5 points give you? There is no hard and fast rule that is scoring reviews. Is there a chart? Is there a reference manual? Is there something that can tell me, the consumer, what the difference is between a 60 and 65? Or a 64, and 65 for that matter? It is broken and doesn't tell you anything. What is a 3 vs a 4? What does a 1 tell you when the game is absolute broken shit? Does a 1 mean it has redeeming qualities? Where is the 0?

Then you have two problems on top of that. People looking at a reviewer's number and basing a decision on it thinking they understand it. Then you have an idiot system called Metracritic, that takes these broken baseless scores, and tries to combine them so people can just look there instead. Then publishers and retailers/digital distribution flash this number around like it is something to consider and be proud of or disappointed by, but nothing to define it.

Scoring systems should be thrown out.

Then there is the issue of gaming reviewers being honest, which they can't without some kind of retribution from whoever supplies the review copies. This latest incident with Duke Nukem Forever is probably only a fraction of actual cases. Remember the game spot reviewer who got canned for bashing Kane & Lynch? Games are scored relatively high to keep metacritic scores higher, and publishers happy. Unless there is some kind of mindset change by publishers and their iron grip on mainstream reviewing, you won't see big changes in the review scores in terms of seeing lower ones. With that in mind, Metacritic will continue to be a tool of evil rather than good.

 

[SL33TBL1ND]

I feel this picture is appropriate:

Spoiler

And I agree. I'm tired of seeing crap get passing scores, and I'm tired of great games being considered crap because their score isn't an 80 or higher.

That makes me even more upset because ilomilo is a pretty fun game, and sadly people will ignore it just because it didn't get in the 90 range.

Yeah, some of the best games I've played are in the sub-80 range.

 

[Jumplion]

I agree with Jim in that Metacritic itself has done no wrong, but it is based on a broken system and it's existence has ultimately done more harm then good in this industry. Whether it is the publishers to blame for interpreting the numbers wrong, or the reviewers for using them wrong, I still stand by my opinion.

See, the thing is that it's not Metacritic's fault that publishers are interpreting these numbers in a detrimental fashion. This speaks of much larger problems in the industry, like how expensive games are getting and how if you don't break 1 million you are screwed. The publishers have caused Metacritic to skew the results, not the other way around. Going for Metacritic is just avoiding the real issue. They do reviews of other mediums and those industries are not as affected by it as this industry is. In a sense, it is kind of like rape-logic, if you will; blaming Metacritic when there's really nothing it can do.

Numbers should not be used. No way, no how. A scoring system for opinion reviewing is the most broken system possible and I cannot understand why it is used. A reviewer could slam a game to hell and back and give it a 60, but then another reviewer can love a different game and have to score it low due to technical problems and score it a 65.

What the hell is the difference? What does 5 points give you? There is no hard and fast rule that is scoring reviews. Is there a chart? Is there a reference manual? Is there something that can tell me, the consumer, what the difference is between a 60 and 65? Or a 64, and 65 for that matter? It is broken and doesn't tell you anything. What is a 3 vs a 4? What does a 1 tell you when the game is absolute broken shit? Does a 1 mean it has redeeming qualities? Where is the 0?

I don't really mind attaching numbers, as I did say I prefer a 5-star rating if numbers must be used. As has been said a million times before, reviews are just purely opinions and putting a number behind it gives a general feel towards how the reviewer felt towards it. It's not an ideal system, but it is a quick and somewhat efficient system. The fact that a 6 means different things to different reviewers is evident that the opinion-based system is working. However, because of the bigger issues of people just looking at the score without reading and comprehending the review, or the fact that an 8 is a "barely passing" grade in this industry, it's gotten to the point where the issues like "what's the difference between a 65 and a 66?" crops up.

Then you have two problems on top of that. People looking at a reviewer's number and basing a decisionon it thinking they understand it. Then you have an idiot system called Metracritic, that takes these broken baseless scores, and tries to combine them so people can just look there instead. Then publishers and retailers/digital distribution flash this number around like it is something to consider and be proud of or disappointed by, but nothing to define it.

Scoring systems should be thrown out.

But again, Metacritic does this kind of thing to movies and stuff and it's not as shit as it is here. It's a good site to get a general opinion on a game. The fact that it is misused by both the public and publishers is just an unfortunate byproduct that Metacritic just can't help. Don't blame Metacritic for doing its job of taking the broken, baseless scores. Blame the people who give out said broken, baseless scores and demand them to fix this kind of shit.

Score systems are not going to be thrown out any time soon, you and I both know that. While hardly an ideal system, it is a somewhat effective one and it helps in general areas. Tossing out the entire concept of scoring just because it is currently abused by publishers and the public is a bit shortsighted and only avoids the main issue of scoring.

Then there is the issue of gaming reviewers being honest, which they can't without some kind of retribution from whoever supplies the review copies. This latest incident with Duke Nukem Forever is probably only a fraction of actual cases. Remember the game spot reviewer who got canned for bashing Kane & Lynch? Games are scored relatively high to keep metacritic scores higher, and publishers happy. Unless there is some kind of mindset change by publishers and their iron grip on mainstream reviewing, you won't see big changes in the review scores in terms of seeing lower ones. With that in mind, Metacritic will continue to be a tool of evil rather than good.

Now, again, this is not Metacritic's fault. Metacritic is not at fault nor is it the core problem. It is simply a tool, neither "good" or "evil". It's just unfortunately misused. We shouldn't be going after Metacritic because some dipshits decided to try and keep their advertisement deals happy, we should be going to the root of all this crap, and Metacritic is not the real issue here, especially in your example.

 

[koroem]

Snip

See this is where you and I are just going to have to agree to disagree. I wholly believe Metacritic is based on a flawed point scoring review system, thus is directly responsible for the negative effects it leaves in its wake. In the end, the people using metacritic use it in a way that may not have been intended, and metacritic itself doesn't appear to have the ability to positively change how it effects everything based around it.

In an ideal world, Metacritic works, but in this real world, it doesn't, and people should not rely on it.

Regardless, you see stuff like this crop up: http://youtu.be/vTemyM_DRq4

Maybe this is an isolated case, maybe not. I'm not about to go through hundreds of reviews looking and doing math to find out. Even one case of it is enough to make me question its reliability on top of everything else I know about how flawed the concept is. Just the fact that this guy, and whoever else this video was directed at take metacritic so seriously as a source for reviews makes me sad.

 

[Xirema]

Personally, I've always sided with Yahtzee on this issue, the perspective being that trying to sum up anything, be it games or movies or books, using a numerical system is inherently flawed. Analysis of art has always depended upon subjective critique, and trying to impose a numerical system on it (numbers being inherently objective) only muddies the discussion.

 

[Andy of Comix Inc]

"Tell me what a three is, versus a two"

'Tis naught but a difference of one!

Well done sir. You have made me LOL IRL, and as result, milk doth flow from thy nose.

 

[Jumplion]

snipers

I'm willing to agree to disagree, but I still don't think Metacritic should get the burden of blame here, at least not most of it. Like I said, it does this for film and is barely matters in that industry. It's not that it's based on a flawed system, it's that they're forced to work with a flawed system. Scoring systems wont go away any time soon, we know that, and they can be used to form a general consensus on the game. It's just that it is skewed so heavily to the 8s and 9s on the scale that it makes the rest of it completely useless and leaves sites like Metacritic and the review system itself to be abused.

Now, yes, there are some flaws with it, mainly how they weight the average and how they aren't entirely open with that process. But we should not be killing the messenger here. We get rid of Metacritic, what then? We're still stuck with a broken scoring system where an 8 is failure, we're still stuck with publishers obsessing over a 86 over an 85, and we're still stuck with many problems that caused this one problem to begin with (expensive hobby, expensive game creation, bad work environment, etc...). It's the leave of a much larger tree of problems, methinks.

 

[Anti-Robot Man]

I like the ten point scale when used properly - with 5 as average. There's no need for reductionism to good/bad, there are good games and excellent games, there are terrible games and merely mediocre. Someone might be intrested in a mediocre game because of a particular mechanic or licence that intrests them, if you crush everything deferentiating just becomes more difficult.

Keeping the score system honest is difficult, reviewers have differing opinions but they are also pressured by the industry and readers. If you ever visit IGN, where anything below a 90 is garbage, the comments sections are filled with bitching over every percentile of difference between the AAA games. Score creep is inevitable, it has to be actively avoided not pandered to, otherwise it just become meaningless. If a site or magazine wants to put things right they need to announce it in a big way that they're rebalancing their reviews (putting 5/50 at average), otherwise the same people will go into a frenzy.

 

[Kross]

Game reviewing has always seemed slightly more generous than movie reviews tend to be. I often find that game magazines seem to focus on whatever is currently hyped with only minor games ever given really bad reviews.

A lot of the issue for smaller outlets is having to buy all the games they review and only having so much time to review games, so they play the ones that seem like more people will be interested in the review of. I know our own editors have traded in many games so they could buy new games to review (usually with their own money - so they typically don't buy games that look like a waste of time). We only recently started getting on enough marketing lists to get a decent chunk of new games sent to us as review copies, but that's only the games with enough budget to do such things.

Steam is great for game reviewers getting access to smaller budget games, as I believe the Steam press accounts just give you access to (near) every game they have. I haven't had a chance to play around on one myself though.

Also, 5 or 2 point systems are the only ones that make sense. Either Good/Bad, or 'unplayable crap'/bad/mediocre/good/awesome - any more granularity in the ranking system other then rating the different technical aspects of the game (sound/video/plot/etc) on these scales is just arbitrary.

 

[geizr]

Don't feel like writing another wall-of-text on this subject, so I'll just summarize my opinion: current numerical scoring of games is a bunch of bogus bullshit.

 

[pppppppppppppppppp]

I never understood what was wrong with the 7-10 rating scale. If I learn a game is good because it's a 9 and bad if it's a 7, it's no different than learning a game is good if it's an 8 and bad if it's a 5. Reviewers are still doing their job either way.

Though on the issue of reviewers giving a bad game good scores to make sure they don't lose advertisement, that's not cool. Personally, I read through several reviews, ask gaming forums or friends, and try to find a demo before I buy a game, a simple metascore isn't enough info to decide how I spend my 60$.

 

[ike42]

I never understood why none of the reviewers set up their scores on a bell curve. It seems that way they would be able to make use of the entire scale. That's how I always do it when I rate anything for my friends. It's very simple. The vast majority of everything falls between 4 and 6. Things that stand out as exceptionally bad or good go towards the tails. What's the point of having 5 (or 50) whole numbers that you never use?

 

[Svenparty]

Game reviewing has always seemed slightly more generous than movie reviews tend to be. I often find that game magazines seem to focus on whatever is currently hyped with only minor games ever given really bad reviews.

A lot of the issue for smaller outlets is having to buy all the games they review and only having so much time to review games, so they play the ones that seem like more people will be interested in the review of. I know our own editors have traded in many games so they could buy new games to review (usually with their own money - so they typically don't buy games that look like a waste of time). We only recently started getting on enough marketing lists to get a decent chunk of new games sent to us as review copies, but that's only the games with enough budget to do such things.

Steam is great for game reviewers getting access to smaller budget games, as I believe the Steam press accounts just give you access to (near) every game they have. I haven't had a chance to play around on one myself though.

Also, 5 or 2 point systems are the only ones that make sense. Either Good/Bad, or 'unplayable crap'/bad/mediocre/good/awesome - any more granularity in the ranking system other then rating the different technical aspects of the game (sound/video/plot/etc) on these scales is just arbitrary.

Thank you for the interesting input and your opinion on the points system. With all the stories that often arise about "fixed reviews" (which I'm sure happens in most mediums) it's interesting to know that smaller companies aren't sometimes part of the review copy lists.

 

[Raziel_Likes_Souls]

I dunno, the system just doesn't work. Considering that every reviewer seems to have a game they're paid to kiss ass to, i.e. IGN kissing up to COD, Greg Tito and Dragon Age 2, etc. It would make this type of system fail. COD gets 10's due to the reviewers getting paid off, and Resonance of Fate gets 7.5 from every critic, and becomes a sequel-less game, despite being miles ahead of Black Cops. Oh man, and don't even get me started on IGN's review of God Hand. Goddamn, I wanted to murder someone after that.