Jimquisition: Review Scores Are Not Evil

[Jimothy Sterling]

The only thing I don't agree in this video is drinking from a recently turd sprayed... water thingy... damn, I need to add a new word to mi English dictionary...

In any case, I don't hate numbers, but I think most people use them just 'cause. The only benefit I see in them numbers, is that I can skip reading a god awful rant about the game and see the bottom line number to get an idea of what the reviewer thinks.

 

[Siege_TF]

i don't really have a problem with scores it's just that they make no sense. first of all i don't believe a complex opinion can be quantified numerically and if it could you would have to decide on a universal scale to use it on, which also makes no sense because different people value things differently.

and there is this weird thing right now where 7/10 is average

5/10 is funtional mechanically, which is not average, despite being the number between zero and ten, because consumers hold the industry to a certain standard. This is mostly thanks to the internet giving consumers the tools to have the developers by their dangly gubbins, and a game that does nothing more than function mechanically (like that X-Men game) won't turn a profit (it didn't). So 7/10 is 'average' in that it meets our standards, as in, it's entertainment that's entertaining, meaning it does more than function, which is what we expect, and have every right to at sixty dollars a pop.

5/10 is a car that runs, 7/10 is a car that runs well, and has air conditioning and a radio. It may not park itself, it may not be a hybrid, it may not have heated seats and a damn GPS, but it's what people consider 'average' in spite of cars not needing A/C or a radio to function.

It's not that complicated.

 

[Guffe]

Cool episode. 8/10
Perfect evil laugh at the end 10/10

 

[veloper]

Any arbitrary grade system works.
A scale from 1 to 11.3 where a 9 is just passable? It still works fine, just aslong as the reviewer is consistent in his grading. Know it and translate to your own scale.

The problem is never the scoring system itself. It's just bad reviews or bad readers.

 

[babinro]

I agree that scores are fine.

The system breaks down over scrutiny but I'd argue that most of us genuinely know this.
We know that a game getting a 9 today doesn't make it superior to every 8.5 game in history regardless of genre.
We know that a 9.5 FPS game isn't necessarily better than a 9.0 FPS on another console, or on a prior console.
We know that one reviewers interpretation of a 7/10 game could vastly differ from anothers.

Numbers provide a quick glance as to whether the article is worth your time. If you've grown familiar with the reviewer, you can better appreciate what the numbers actually mean in terms of game strength.

 

[Mrkillhappy]

Review scores by themselves can be helpful, if I just want a quick overview of whether or not a game is good however the scores should give some for of scale with them. For example let us know what qualifies a mediocre and what is considered good this can lead to a lot less fanboy complaints.

 

[wolfyrik]

Wait, what? This is an issue? People who complain about review scores, get 3/10.
I'm giving them a point for being able to string a coherent sentence together, or at least enough of one that we can understand what they're complaining about. They also receive a further two points for having the gall to express themselves on the internet.

 

[sindremaster]

So you missed the most obvious point? How about the fact that scores provide zero information! None! Nothing!

They tell you how much the reviewer liked the game. It doesn't tell you why, but that's what the text is there for.

 

[Mahoshonen]

I kinda get how 7/10 became average if you think about it like a school grading system. A "C" is meant to be the average grade. By the same idea, 7/10 means you'll get you'll most likely get your money's worth from the game, but no more. The big issue that's brought up about scores, that they're subjective, is a problem inheiret with reviews themselves, and removing scores won't mitigate this problem.

 

[Mythmaker]

True, Review Scores in and of themselves aren't bad. But there is a fundamental problem with rating a piece of entertainment like a consumer product. While a consumer product is usually meant to be replaced, entertainment is different. If you rated a movie when it came out in theaters, should you give it a second score when it released on Blu-Ray? No, because the quality of the piece doesn't change, just its value. Quality and value aren't the same thing, and the industry at large really needs to acknowledge that, IMO.

If the industry wants scores to mean something, they need to change two things.

First, the length of a game shouldn't affect the review score by itself. Sure, $60 for a 5-hour game probably isn't worth it, but that has nothing to do with the quality of the game. If something like Portal, for instance, were released as a stand-alone $60 game, should it be considered a lesser game than if it was sold for $10? Letting price factor into a review of entertainment also damages the score by dating it; in a year, a game can drop to half its price. Should it be considered a better game if it does? If people really want to address price, they should do it, but not as part of a score. Set a price limit, or a recommended format, but don't consider it in your score unless the length, or lack thereof, affects the quality of the game itself, not its value.

And second, the review scales should not use other games as their base. This might sound stupid, so let me give an example. New Super Mario Bros released more than 3 years ago, and got scores in the high 80's. New Super Mario Bros 2 released this year, with scores in the high 70's. Most (but not all) reviews I've seen credit the lower score to the game being too much like its predecessor. What does that have to do with the game's quality? Nothing. It has everything to do with the game's value. I see the same thing when games are compared to one another. A game's quality shouldn't be dependent on what its competitors are doing, but on its own merits. This also dates the review further, because its dependent on games that were out at the time of the review. Its value should be what's affected, not its quality.

Now I'm not saying that other games shouldn't affect scores at all; in fact, by defining a critic's quality spectrum, they already are. But the level of engagement should be what's scored, not how much the game is worth. If they want to include that, they should score the game's value separately.

 

[muffinatorXII]

i don't really have a problem with scores it's just that they make no sense. first of all i don't believe a complex opinion can be quantified numerically and if it could you would have to decide on a universal scale to use it on, which also makes no sense because different people value things differently.

and there is this weird thing right now where 7/10 is average

5/10 is funtional mechanically, which is not average, despite being the number between zero and ten, because consumers hold the industry to a certain standard. This is mostly thanks to the internet giving consumers the tools to have the developers by their dangly gubbins, and a game that does nothing more than function mechanically (like that X-Men game) won't turn a profit (it didn't). So 7/10 is 'average' in that it meets our standards, as in, it's entertainment that's entertaining, meaning it does more than function, which is what we expect, and have every right to at sixty dollars a pop.

5/10 is a car that runs, 7/10 is a car that runs well, and has air conditioning and a radio. It may not park itself, it may not be a hybrid, it may not have heated seats and a damn GPS, but it's what people consider 'average' in spite of cars not needing A/C or a radio to function.

It's not that complicated.

i disagree, 5/10 should be a game that isn't bad but isn't great, a game that you wouldn't mind playing but you wouldn't actively seek to play it. a 7/10 should be a good game, one that you would seek to play if you can (maybe buy it on sale because it looks cool) but probably wouldn't get in line for a day for it.

but anyway my main point still stands, review scores make no sense

 

[GonzoGamer]

I don't care when games get underrated.
It bothers me when they're overrated, which is usually the case.
It's the reason I rent: after everyone said No More Heroes was amazing and I wasted money on that piece of tedious crap. Sure some reviews are just plain entertaining but I don't know if I'm going to like a game until I've tried it.

 

[jmarquiso]

Scores, no problem with them. Now Metacritic can die in the pits of 8th level hell for the damage that has done.

For some reason, people are mixing the hate of Metacritic with scores. That is a very bad idea to muddle up on your stance. Many of us know that Metacritic have costed people jobs, not intentionally, but companies have relied on using it and thus have seen it as a way to see how well their product will or will not do.

Scores of an individual and the text for why that score exist, is fine. Throwing several different scores formats, and trying to find an average, along with customer scores is idiotic. It makes little to no sense and it's hard for people figure out or analyze why one game got mostly 8/7 score and some twat gave it a 2/5 stars.

Metacritic didn't cost people jobs. Publishers with terrible policies and stupid expectations have cost people jobs. Gabe Newall's talked about it as a stupid way to motivate workers - it causes people to latch on to winning properties (i.e. more of the same) rather than actually attempt new things, thus creating new properties. All metacritic is is an aggregation service. Hating on it is like hating on Google Reader.

 

[jmarquiso]

Am I the only person who thought of Raif Feinnes during Schindler's list in the last few minutes? That was about dirt under the fingernails, so it didn't work QUITE as well.

 

[Icehearted]

Awe, there was a nut in that turd, wasn't there? :\

I have absolutely no idea where this ire for scores of 8 and below came from, or what kind of twit gets mad about a score. I've seen it happen, I've seen people try to drum up outrage because a game got a score of less than 9/10, but I never made sense of why it mattered to them or anyone else. I don't get the personal offense here.

All of this aside however, there was a nut in that turd...

 

[ConanThe3rd]

You call that a "Wrong!", Jim?!

 

[Sir Thomas Sean Connery]

At the end, I could feel Jim's raging dictator boner through the screen.....

I tend to agree. Scores reflect an opinion and can be quite fun.

What needs to be stopped is Metacritic and other sites that publishers put so much weight on. Yeah, it would be best if they just didn't, but as long as it exists, they will.

 

[geizr]

There's nothing wrong with a review score. There is something wrong with how review scores are often abused, and it's not just readers misunderstanding the context of the review score. It's also reviewers that have, over time, whether deliberated or accidentally, distorted the meaning of the score such that only a small span of the entire range are given meaning. The Hate out of Ten problem comes partially at the fault of reviewers who have, over the years, created the distortion that scores in the range of 7.0-8.0 are only average or mediocre, as opposed to the 4.5-5.5 range having that meaning. Back in the day, if a game was truly shitty, it got a 3/10, at best. Today, that same level of shittiness garners a 6.5-7.0, a range of scores that was once considered slightly above average in quality. Even further, a game having such a score would still often appeal to fans of the game's genre; I'm not sure if the same holds true today.

That segues into another basic problem that I perceive regarding the review score; it lacks context of personal preference. Just about every review score seems to be given an absolute context when the reality is that a game that is considered shitty or mediocre by one individual may be considered exceptional or superior by another individual. The difference is personal taste and preference. Review scores often don't take into account the skews in perception of quality that depend on the preferences of the target audience. Fans of a particular genre or series are more likely to have a more elevated opinion of a game in their preferred genre or series, whereas those who dislike a particular genre or series will have a much low opinion of the game. Those who are ambivalent are likely to be somewhere in the middle. No singular review score is going to properly capture this nuance.

It is for this reason that I think review score should really be done in triplet, one score for fans, one score for haters, and one score for the general ambivalent populous. Of course, this means you have to have 3 different reviewers for any given game and they must have the particular dispositions toward the game as required by the particular type of score they are trying to yield. The primary problem I see with this solution is that it is expensive (for the reviewing publication) and cumbersome (it's hard finding that mix of fan, hater, and "don't care" attitudes), meaning the review process slows significantly and can cost publications looking for rapid turn-around. However, in my opinion, this would allow a lot more context to be applied to the game and give a more complete view of the game's quality.

 

[veloper]

First, the length of a game shouldn't affect the review score by itself. Sure, $60 for a 5-hour game probably isn't worth it, but that has nothing to do with the quality of the game. If something like Portal, for instance, were released as a stand-alone $60 game, should it be considered a lesser game than if it was sold for $10? Letting price factor into a review of entertainment also damages the score by dating it; in a year, a game can drop to half its price. Should it be considered a better game if it does? If people really want to address price, they should do it, but not as part of a score. Set a price limit, or a recommended format, but don't consider it in your score unless the length, or lack thereof, affects the quality of the game itself, not its value.

This is a very good point. Prices drop all the time, especially on PC and length is only value for money (if the game is any good at all).
Game length should be mentioned somewhere, but it doesn't make an experience more fun. Some of us don't even want to waste much time.

And second, the review scales should not use other games as their base. This might sound stupid, so let me give an example. New Super Mario Bros released more than 3 years ago, and got scores in the high 80's. New Super Mario Bros 2 released this year, with scores in the high 70's. Most (but not all) reviews I've seen credit the lower score to the game being too much like its predecessor. What does that have to do with the game's quality? Nothing. It has everything to do with the game's value. I see the same thing when games are compared to one another. A game's quality shouldn't be dependent on what its competitors are doing, but on its own merits. This also dates the review further, because its dependent on games that were out at the time of the review. Its value should be what's affected, not its quality.

Now I'm not saying that other games shouldn't affect scores at all; in fact, by defining a critic's quality spectrum, they already are. But the level of engagement should be what's scored, not how much the game is worth. If they want to include that, they should score the game's value separately.

Trickier. A reviewer may want to reward originality somehow, but at the same time to a newcomer who isn't jaded yet, copy-pasta sequel X may still be worth their time.
Worse, a fresh experience may genuinely make the critic like the experience more. If this rule is to be taken to it's logical destination, then a critic would have to go something like: 'wow I loved this game! Hmm, maybe it's because it's something not done before, so I'd better substract a full point just to be fair to all the copycats that will soon follow'.

 

[Imp_Emissary]

Well done Jim. xD The end of the show killed me! HA!

I don't see how people blame review scores or Metacritic for the choices of the people who run a company they aren't a part of. Even if review scores and metacritic left, or weren't ever there to begain with the people at the top of these companies would just make up some other bullcrap reason to fire people.

Also, those gloves do look better.