Is game reviewing 'broken' as a system?

[Treblaine]

Actually, systems such as reviews on scale of 100 (they are all scales of 100 basically, even on a 1-10 scale because they use to the tenths decimal place) tend to be totally subjective. The metric is, in and of itself, flawed. Leonard Mlodinow explains it beautifully in the book, The Drunkard's Walk. Essentially, people have tried to create a system such as this for grading papers for school. This one fella' took months drilling strict criteria for his out of 100 scale paper grading system. He trained 25 people. After the training, he gave them all the same stack of 25 papers to grade. When he looked at the results of his work, peoples grades varied by as much as 12 points on the same paper. He also tells the story of how two of his son's friends turned in identical papers, accidentally. Instead of getting caught, one received a 90 while the other received a 79. For the number system to mean anything, there would have to be a palpable difference from one point to the next, but there isn't.

The margin for error is huge. I would invite you to check out metacritic (I know, I hate it too, but it shows my point beautifully). In the official reviews column on a game that has seen international release, there tends to be a huge difference in scores out of 100. One game (it's been a while so I don't remember the game) had a top score of 71, while the lowest score on the list for the game was a 10.

Also, I know no one is a fan of a by the numbers system.

But isn't the point of Metacritic to smooth out that margin of error?

Like how a junk rifle can be inaccurate, keep firing it at the target and the centre of the cluster of bullet holes will be actually your centre of aim.

You give the perfect example with your paper marking to the inaccuracy between two shots of a rifle, the identical paper marked by different people, to get the true score you have it mark it by many more teachers and the most accurate score is the average of all their scores.

Metascore. THAT is what we are talking about here. Frankly, critics might as well never reveal publicly their score to a game, they should send it straight to metacritic to find the aggregate. As a single score in itself is useless due to the inaccuracies in trying to quantify ones judgement. And it doesn't do any good as then the fanboys and haters say "GRRR, IGN skewed the result, if it wasn't for them this game would have had a different score! GRRR"

The worth of a single critic should be in their prose. What they actually write about a given work, that is the most important guide to the customer.

As to Modern Warfare 3, consider this: it may be hardly an improvement over COD4 but:
-COD4 is still a good game, 4 years later
-No other game has really surpassed it in what it does.

So, standards have NOT gone up significantly, MW3 is a bit better than COD4 in the most valued areas and enough things are changed around for it to get the same score as COD4. They may rate it a bit higher, but another critic will rate it a bit lower. THERE IS NO REASON FOR CONFORMITY! People can have varying opinions and judgements.

 

[jacilon]

We should raise the max score to one million.

 

[masticina]

Mmm it is not easy I do agree that it seems like the gamereview industry has been pulling up allot.

Self? Well Self I am way thougher on games then some review websites. Of course we know the japanese "Oh it is a final fantasy 10/10" and the "The new Halo is definitely one of the best experiences this year" of the western market.
Pay
But some websites actually dare to be negative. Giantbomb for instance it does gives at times 2 out of 5 stars "Meh" and yes also 4 our of five stars.

Reviewing is subjective anyway and I would agree that a 10/10 is possible even for a game with flaws. IF those flaws are inconsequential compared to what works. If there are 3-4 flaws on 250 things done well. You know then it is a 10!

In the end I read reviews.. yeah I read them, I watch gameplay videos I make up my own mind. Is Halo a great shooter? Probably but I don't do shooters so I could care less.

Do I find that bringing out a new title every year with only small progression is cheap? Yes! Paying full price for a few small updates or addition of a few more maps ... seriously

So in the end I will say that I rather read reviews then go on by their numbers. I mean I have enough games that we're "panned" by giving a 7.5. I as player can look through those flaws and enjoy what works.

Take The Saboteur not the highest scoring game, neither without flaws, but it is fun! Very Very Fun!

 

[Daveman]

I dunno, I take 9-10 as an instant classic, 7-8 as good regardless, 5-6 as good if you enjoy that genre and anything less as pretty poor.

 

[jonyboy13]

The system isn't defined as it should. 7\10 could mean that the graphics are shit but the gameplay and story are amazing. It could also mean that the gameplay is annoying, repetitive and lots of QTE.
The true problem though, is the reviewers themselves. Some don't know the difference between a fact and opinion and others just get paid to stfu about the bad stuff in a certain game. It's easily done since if they say something bad they piss off a huge publisher who will just boycott them, losing the site a lot of money.

At this point, I don't bother looking at most reviews. I just watch some gameplay on youtube and judge it by myself. Hopefully, one day reviewers won't be biased idiots and their sites won't be threatened by huge publishers.

 

[Pedro The Hutt]

You know, if you're so upset about the scoring, just read the whole damn review, that should tell you you need to know about what the reviewer thinks of the game. =P

That said, with some games having two groups of raving lunatics, one hating the game and the other loving it even though neither have ever played it (*cough*CoD/BF*cough*), game reviewing can be pretty ungrateful business. If you score it a 7.5/10 no matter what you think of it or what your text says, the fans will send you death threats for denying it its god given right to a 9 or 10/10 score, or accuse you of accepting bribes from the a publisher for skewing the score.
And if you give it a 9/10, even if it's because you genuinely and really enjoy the game, the haters will send you death threats and/or call you a pandering hack, a biased idiot, a mindless fanboy, or you get accused of taking bribes from the game's publisher in exchange for inflating the score.

So really, with some games, you're going to get flak and hate from one of those two parties no matter what you score it.

I'd say that these rabid fans can be just as much at fault for influencing a review's score as a publisher, if not even more so. The publisher won't spam the reviewer's inbox with some of the most hateful bile ever committed to text if the review isn't to their liking. And having worked for a medium sized Belgian review site myself, odds are that even despite less than favourable reviews, if the criticism is well founded, well described and with merit, they'll continue to send you review copies anyway.

 

[Soviet Steve]

The whole concept of reviewing something is flawed because reviews are always going to be subjective - You can argue the rational points and recount what is possible within the game but otherwise everyone takes something different from each title.

 

[RedEyesBlackGamer]

Games are reviewed with words, not numbers. If people didn't put so much emphasis on the score then there wouldn't be an issue.

As for scoring being broken, it varies from publication to publication.

This. The actual reviews are still there, so it really doesn't matter. As for the number on the end, yes, there has been inflation. AAA titles get special treatment in this regard, as admitted by a reviewer on this site.

 

[xDarc]

I don't agree with metascores on the internet for the same reason ron paul will not be the republican nominee even though he blows all the other candidates out of the water on the web.

The internet is just a bloated, biased ball of shit.

I long for the days when a panel of reviewers from a mag each scored a game, and you knew the reviewers personalities in a sense, identifying with one that best matched your outlook- and then weighted his scored against the rest.

In my mind a game like MW3 is a 6.5 at best because of all the recycled content.

 

[Deathninja19]

No.

I think it's a hijacked system with misconstrued averages, fans who don't have a fucking life and marketing taking quotes and awards out of context.

Internet high five to this guy here.

 

[Plinglebob]

While I agree that most the time the system for games is broken, I disagree with anyone who says that games shouldn't have scores or that no game should get a perfect score. Other mediums have had reviews with scores for decades and very few have an issue with it.

My personal, completely unscientific, view is the biggest problem (and the reason for the high score bias) is that a lot of people who review games are fanboys who have been given a megaphone. While this is true at one end of any group of critics, other mediums have the additional benefit of two important things. The first is an old guard. Gaming desperately needs its Barry Norman's and Roger Ebert's fast. People who have been reviewing games for years and so are able to give an un-biased opinion. The second is no-one can agree what a "Good Game" consists of. In other mediums there is a general concensus in what makes a film/book/painting "Good" ranging from Story to Acting to Cinematography to Brushwork to Story Structure. Games don't have that. Hopefully, both of these will come in time, but until then just pick some outlets who's views you trust and try and read between the lines of what they say.

 

[Baresark]

snip

But isn't the point of Metacritic to smooth out that margin of error?

Like how a junk rifle can be inaccurate, keep firing it at the target and the centre of the cluster of bullet holes will be actually your centre of aim.

You give the perfect example with your paper marking to the inaccuracy between two shots of a rifle, the identical paper marked by different people, to get the true score you have it mark it by many more teachers and the most accurate score is the average of all their scores.

Metascore. THAT is what we are talking about here. Frankly, critics might as well never reveal publicly their score to a game, they should send it straight to metacritic to find the aggregate. As a single score in itself is useless due to the inaccuracies in trying to quantify ones judgement. And it doesn't do any good as then the fanboys and haters say "GRRR, IGN skewed the result, if it wasn't for them this game would have had a different score! GRRR"

The worth of a single critic should be in their prose. What they actually write about a given work, that is the most important guide to the customer.

As to Modern Warfare 3, consider this: it may be hardly an improvement over COD4 but:
-COD4 is still a good game, 4 years later
-No other game has really surpassed it in what it does.

So, standards have NOT gone up significantly, MW3 is a bit better than COD4 in the most valued areas and enough things are changed around for it to get the same score as COD4. They may rate it a bit higher, but another critic will rate it a bit lower. THERE IS NO REASON FOR CONFORMITY! People can have varying opinions and judgements.

I agree that the value of a review is in the actual prose describing what the critic thinks of his/her experience. But, I think doing away with a number system altogether would the favorable thing to do. Let their only be the contents of the review, and not the completely arbitrary number scale that reviewers like to use.

The reason metacritic does things they way they do is because you can't possibly trust the aggregate data without seeing the constituents for yourself. If all people submitted a score to metacritic and the mean was available for people to see, it wouldn't tell the whole story. If you think the accusations of bought reviews are bad now, you wouldn't believe what they would be if you only saw the aggregate number. There would simply be one person to pay off, rather than ten. I have more often than not bought and very much enjoyed a game that has received a 7.0/70 on the numbering system. But, I always read the reviews and see why it got the score it did.

Unfortunately, when you get to the upper tier scores, the number is the most concentrated thing. For instance: a reviewer may think, after the initial play through of a game, that the game deserves a 95/9.5. But, the score shouldn't be based on initial feelings about the game, but readable data. How are the controls? Did the game keep you engaged? Was the UI conducive to what the player would need to enjoy the game? I have heard people like Yahtzee explain it best (there is a scary thought). When you played the game, even if you loved the experience, you need to look at it outside your feelings about the game to the hard facts of the game. Take his Arkham City review. He gave the game a recommend, but he wasn't afraid to say what the game did wrong (in his opinion). For someone, the things it did wrong may detract from the game enough where maybe it wouldn't deserve the praise it got in the first place. Not that I have a problem with Arkham City, I very much look forward to the PC release.

Also, all those things that you said about MW3 versus MW are true. The only problem is that the reviewers (in general) harp on other games and lower the score when it hasn't been enhanced over it's predecessors while using the same game mechanics. That is not an objective view of things at all. I think that the numbered score is the biggest worry for both companies and game reviewers. Most likely because a positive metacritic score looks best for both parties. I'm sure we all remember the Dragon Age 2 debacle. I liked the game, not as much as the first one, and I waited for a good sale on Amazon to get it. But most people think the game did not deserve the high praise it had gotten. And the focus of the praise was it's number score.

Also, the most important way of testing your number metric is to see if it fits into a "bell curve". This is the way that statisticians check the margin of error. Usually though, nothing on metacritic makes a proper bell curve, at least not what I have seen.

Edit: Also, one of the reasons why metacritic fails to perform on the bell curve test is because there simply aren't enough reviews. Another good reason to do away with the number system.

 

[aegix drakan_v1legacy]

The system's not broken. It's just shit to begin with.

Personally, I don't think you can measure quality and put a definitive number on it and it also seems to discourage people from actually reading the review to see what is good and bad about a game.

The number rating system is simply accommodating laziness.

This. 100% this. And yet, my engineering friend keeps asking me to grade games I play with a number scheme, and doesn't agree with me that the concept is stupid.

You CAN'T apply an objective number to a subjective opinion. It's a stupid concept, and as the quoted guy said, it simply encourages and accommodates laziness. Oh, and it leads to scores being ratcheted up because of fanboys whining how certain games "only" got a 9.

Personally, I grade games on a scale of: Avoid - Decent - Solid - Fantastic. Simple and effective.

 

[DustyDrB]

1) Perfect score =/= a perfect game.

2) Reviews are opinions.

3) Are reviewers paid off? Maybe. Maybe not. I go with saying they are innocent until proven guilty, and don't throw around that accusation like a jackass.

4) On inflated scores: The games that would be getting a ton of ratings from 1-5 on a scale of 10 are just not being reviewed. How many outlets waste their time reviewing the latest shovelware? How many sites spend that time reviewing a game they know will likely be of the lowest quality, and a game that their viewers really don't care about? And the most broken, unplayable games out there almost always cancelled.

5) Despite me commenting on scoring, I think it is a system we would be better off without. Art (or entertainment) is reviewed with words. It's about describing the experience. Numbers don't do that. The scoring system has deluded many people into thinking there is some absolute and objective measure to judging games when there is no such thing.

 

[Treblaine]

In my mind a game like MW3 is a 6.5 at best because of all the recycled content.

Yes, recycled content, but recycled content from a very very good game. COD4 garnered universal praise and no one had objection to that.

Points deducted for lack of originality, yes, but 30% off for that?

Half Life 2: Episode 2 came out so many years after HL2 with no new weapons nor new enemies, but scored a 90%, should it have scored 65%?!? Sure it was sold at a low price but MW3 adds a lot more to the core experience (the multiplayer).

And really, it's not like a game has come along that does the same thing as COD4 (fusion of RPG mechanics with online FPS competitive gameplay and game-changing killstreaks with focus on tight close quarters combat).

Tomb Raider to Tomb Raider: Last Revelations Over 4 years. It's the same engine, premise, very similar weapons and gameplay. Both got very similarly high scores.

People who didn't like Tomb Raider 1 bashed Tomb Raider mercilessly for how its "just the same game with a few small changes" but fans of Tomb Raider were happy to get a new adventure every year.

I see the same thing here, people who don't like COD (including COD4) begrudgingly accept praise for COD4 yet play the originality card when the series has continued success, as if originality REALLY made up 30% of a review score. To fans of the series - the people who would actually buy the game, the people who would actually read the reviews - then it being a lot like COD4 is a good thing.

 

[Slycne]

So I have a theory that's not often explored whenever this discussion of review score inflation comes up.

I think games are simply better these days or at least what we consider to be an average experience. With budgets in the millions of dollars, it's not often that we see Big Mother Trucker level of terrible being released now.

As with many things, I think the answer probably actually somewhere at the intersection of everything presented thus far.

 

[Treblaine]

snip

But isn't the point of Metacritic to smooth out that margin of error?

Like how a junk rifle can be inaccurate, keep firing it at the target and the centre of the cluster of bullet holes will be actually your centre of aim.

You give the perfect example with your paper marking to the inaccuracy between two shots of a rifle, the identical paper marked by different people, to get the true score you have it mark it by many more teachers and the most accurate score is the average of all their scores.

Metascore. THAT is what we are talking about here. Frankly, critics might as well never reveal publicly their score to a game, they should send it straight to metacritic to find the aggregate. As a single score in itself is useless due to the inaccuracies in trying to quantify ones judgement. And it doesn't do any good as then the fanboys and haters say "GRRR, IGN skewed the result, if it wasn't for them this game would have had a different score! GRRR"

The worth of a single critic should be in their prose. What they actually write about a given work, that is the most important guide to the customer.

As to Modern Warfare 3, consider this: it may be hardly an improvement over COD4 but:
-COD4 is still a good game, 4 years later
-No other game has really surpassed it in what it does.

So, standards have NOT gone up significantly, MW3 is a bit better than COD4 in the most valued areas and enough things are changed around for it to get the same score as COD4. They may rate it a bit higher, but another critic will rate it a bit lower. THERE IS NO REASON FOR CONFORMITY! People can have varying opinions and judgements.

I agree that the value of a review is in the actual prose describing what the critic thinks of his/her experience. But, I think doing away with a number system altogether would the favorable thing to do. Let their only be the contents of the review, and not the completely arbitrary number scale that reviewers like to use.

The reason metacritic does things they way they do is because you can't possibly trust the aggregate data without seeing the constituents for yourself. If all people submitted a score to metacritic and the mean was available for people to see, it wouldn't tell the whole story. If you think the accusations of bought reviews are bad now, you wouldn't believe what they would be if you only saw the aggregate number. There would simply be one person to pay off, rather than ten. I have more often than not bought and very much enjoyed a game that has received a 7.0/70 on the numbering system. But, I always read the reviews and see why it got the score it did.

Unfortunately, when you get to the upper tier scores, the number is the most concentrated thing. For instance: a reviewer may think, after the initial play through of a game, that the game deserves a 95/9.5. But, the score shouldn't be based on initial feelings about the game, but readable data. How are the controls? Did the game keep you engaged? Was the UI conducive to what the player would need to enjoy the game? I have heard people like Yahtzee explain it best (there is a scary thought). When you played the game, even if you loved the experience, you need to look at it outside your feelings about the game to the hard facts of the game. Take his Arkham City review. He gave the game a recommend, but he wasn't afraid to say what the game did wrong (in his opinion). For someone, the things it did wrong may detract from the game enough where maybe it wouldn't deserve the praise it got in the first place. Not that I have a problem with Arkham City, I very much look forward to the PC release.

Also, all those things that you said about MW3 versus MW are true. The only problem is that the reviewers (in general) harp on other games and lower the score when it hasn't been enhanced over it's predecessors while using the same game mechanics. That is not an objective view of things at all. I think that the numbered score is the biggest worry for both companies and game reviewers. Most likely because a positive metacritic score looks best for both parties. I'm sure we all remember the Dragon Age 2 debacle. I liked the game, not as much as the first one, and I waited for a good sale on Amazon to get it. But most people think the game did not deserve the high praise it had gotten. And the focus of the praise was it's number score.

Also, the most important way of testing your number metric is to see if it fits into a "bell curve". This is the way that statisticians check the margin of error. Usually though, nothing on metacritic makes a proper bell curve, at least not what I have seen.

Edit: Also, one of the reasons why metacritic fails to perform on the bell curve test is because there simply aren't enough reviews. Another good reason to do away with the number system.

I have also played 7.0 games and thoroughly enjoyed them and you know what they are still 7.0 games and they always will be

Because i think you fundamentally misunderstand what the score actually is. It IS NOT an absolute measurement of a games worth. What is is the the consensus of the industry AT THAT TIME!!

You may enjoy a game because - get ready to have your cranium exploded with this intensely profound revelation - you do not perfectly conform with consensus! You CAN HAVE A DIFFERENT OPINION! And you know what, the rest of the industry does NOT have to file up in line behind your opinion.

Example, I played Blood Omen 2, a typical 7.0 kinda game. It had stiff controls, it was repetitive, and meandering. Every critisciem about it was well deserved, the consensus of the industry is that it wasn't a very good game. But I loved it, because in this case I could deal with the controls and what it did offer was worth it to me. Personally, I'd give it a 9.0 but I'd never suggest any critics was either disingenuous or was unsuited to review such a game if they gave it a 7.0/10.

7.0 is well deserved as I know the average gamer would not tolerate the controls and glitches I had to deal with, and the appeal of being a badass vampire might be poor substitute.

Modern Warfare is NOT the only franchise that keeps high scores to spite only small iterative improvements. What about:

-Assassin's Creed
-Tomb Raider (classic)
-Zelda
-Mario Galaxy 1 & 2
-Gears of War (How has Gears 3 REALLY changed from Gears 1?)

Critics ARE going to talk about lack of originality, they are in this business to see progress. But when it comes down to it a good game is a good game. See critics don't REALLY deduct points for lack of originality. They talk about it but they don't. You know what, you can't re-invent the wheel every 12 months. I think the problem with being unoriginal is not continuous but it suddenly becomes intolerable WHEN... a better competition comes along. Nothing has come along to REALLY competes with COD, it wasn't BF3 with with its vehicle focus and doesn't have the RPG elements and crazy killstreaks that make COD multiplayer so engaging.

Some of the most applauded works of film and video games have been incredibly original. Star Wars is famously an aggregate of dozens of different sources, Fistful of Dollars a remake of a Japanese samurai film, Indiana Jones is often cut for cut remakes of those 1930's adventure-serials.

The reality is you can only have so many review for one product and face it, it's not like we are testing life changing drugs for clinical trails, you have to follow a rule of thumb on these sorts of things the maths exists to come up with a definitive score with only 20-30 reviews. And this sort of thing has to be done all the time such as a rifle being sighted in the point of aim is found with a group of only 5 shots, and that's sighted in enough to hit a target over a mile away. You don't need a perfect normal-distribution to say something vaguely conclusive.

Also it's pointless trying to be too precise as the system is self-aware, the metascore itself feeds back on further scores!

Metacritic is a rough guide and face it, as much as the bean counters may use metacritic when all the money has been counted it is the actual REVENUE that counts!

 

[DustyDrB]

So I have a theory that's not often explored whenever this discussion of review score inflation comes up.

I think games are simply better these days or at least what we consider to be an average experience. With budgets in the millions of dollars, it's not often that we see Big Mother Trucker level of terrible being released now.

As with many things, I think the answer probably actually somewhere at the intersection of everything presented thus far.

That's part of a point I've tried to make in several of these threads. You guys just don't review the terribad games because it's not worth your time (your readers probably don't care about the latest shovelware) or because those games don't make it very far in development before they are cancelled.

 

[Red Roark]

Personally, I think it should be like schools grade papers . 9.0-10.0 = A(near perfect), 8.0-8.9= B(Very Good), 7.0-7.9= C(Good), 6.0-6.9= D(Okay), 5.0-5.9= F(Not good), <4.9= Why does this exist?

If you like some aspects of the work in question, and feel that if done a certain way could have really salvaged the experience. Ie: wasted potential. Say you're reading a comic and you really like the villain and would love to see more about him and his involvement in the story line. They did something right to get you intrigued so that's positive points for it. But they didn't follow through and make his motivations clear and he continues to get little screen time. Or maybe that aspect is going well and he has clear motivations but he does something copletely out of character at the end that throws you out of the experience or makes the ending contrast with themes of the story so far (and it's possible to do that really well too, for a good example see the original ending in Watchmen the graphic novel. Really you can pull off anything it's more about execution.)

That's when I bust out my <4.9