Treblaine said:
Baresark said:
But isn't the point of Metacritic to smooth out that margin of error?
Like how a junk rifle can be inaccurate, keep firing it at the target and the centre of the cluster of bullet holes will be actually your centre of aim.
You give the perfect example with your paper marking to the inaccuracy between two shots of a rifle, the identical paper marked by different people, to get the true score you have it mark it by many more teachers and the most accurate score is the average of all their scores.
Metascore. THAT is what we are talking about here. Frankly, critics might as well never reveal publicly their score to a game, they should send it straight to metacritic to find the aggregate. As a single score in itself is useless due to the inaccuracies in trying to quantify ones judgement. And it doesn't do any good as then the fanboys and haters say "GRRR, IGN skewed the result, if it wasn't for them this game would have had a different score! GRRR"
The worth of a single critic should be in their prose. What they actually write about a given work, that is the most important guide to the customer.
As to Modern Warfare 3, consider this: it may be hardly an improvement over COD4 but:
-COD4 is still a good game, 4 years later
-No other game has really surpassed it in what it does.
So, standards have NOT gone up significantly, MW3 is a bit better than COD4 in the most valued areas and enough things are changed around for it to get the same score as COD4. They may rate it a bit higher, but another critic will rate it a bit lower. THERE IS NO REASON FOR CONFORMITY! People can have varying opinions and judgements.
I agree that the value of a review is in the actual prose describing what the critic thinks of his/her experience. But, I think doing away with a number system altogether would the favorable thing to do. Let their only be the contents of the review, and not the completely arbitrary number scale that reviewers like to use.
The reason metacritic does things they way they do is because you can't possibly trust the aggregate data without seeing the constituents for yourself. If all people submitted a score to metacritic and the mean was available for people to see, it wouldn't tell the whole story. If you think the accusations of bought reviews are bad now, you wouldn't believe what they would be if you only saw the aggregate number. There would simply be one person to pay off, rather than ten. I have more often than not bought and very much enjoyed a game that has received a 7.0/70 on the numbering system. But, I always read the reviews and see why it got the score it did.
Unfortunately, when you get to the upper tier scores, the number is the most concentrated thing. For instance: a reviewer may think, after the initial play through of a game, that the game deserves a 95/9.5. But, the score shouldn't be based on initial feelings about the game, but readable data. How are the controls? Did the game keep you engaged? Was the UI conducive to what the player would need to enjoy the game? I have heard people like Yahtzee explain it best (there is a scary thought). When you played the game, even if you loved the experience, you need to look at it outside your feelings about the game to the hard facts of the game. Take his Arkham City review. He gave the game a recommend, but he wasn't afraid to say what the game did wrong (in his opinion). For someone, the things it did wrong may detract from the game enough where maybe it wouldn't deserve the praise it got in the first place. Not that I have a problem with Arkham City, I very much look forward to the PC release.
Also, all those things that you said about MW3 versus MW are true. The only problem is that the reviewers (in general) harp on other games and lower the score when it hasn't been enhanced over it's predecessors while using the same game mechanics. That is not an objective view of things at all. I think that the numbered score is the biggest worry for both companies and game reviewers. Most likely because a positive metacritic score looks best for both parties. I'm sure we all remember the Dragon Age 2 debacle. I liked the game, not as much as the first one, and I waited for a good sale on Amazon to get it. But most people think the game did not deserve the high praise it had gotten. And the focus of the praise was it's number score.
Also, the most important way of testing your number metric is to see if it fits into a "bell curve". This is the way that statisticians check the margin of error. Usually though, nothing on metacritic makes a proper bell curve, at least not what I have seen.
Edit: Also, one of the reasons why metacritic fails to perform on the bell curve test is because there simply aren't enough reviews. Another good reason to do away with the number system.