Review Scores are Shit

[Lizzy Finnegan]

Review Scores are Shit

With no industry-wide standard for review scores, are scores themselves - and to a larger degree, aggregate sites - beneficial? The short answer is no.

Read Full Article

 

[Neurotic Void Melody]

It's a holiday, not a vacation...what flippant destruction of the English language you impose upon us all? If the pen is truly mightier than the sword, then thou hast cleft my heart in twain! Also may have unleashed a toxic genocide on some hypothetical illiterate orphans. 1/10. And no sticker stars. Clumsy American slangwangles.

Anyhow, agreeable article. It's a pity it needs to be said really. In the last couple of years of intergrating into online culture, it never ceases to amaze me how attached people can get to their idea of a number to label a subjective work of...art?...work? How about not using numbers or stars and instead rate media using a chart of various layered textures? That'll throw them off balance! Or a selection of well known chocolates. COD Black Ops 3 can be a Kitkat chunky. Wolfenstein TNO will be assigned a Terry's chocolate orange with a cup of Earl Grey tea. Yup, beverages can be used too! Uncharted 4 gets a big selection tub of Roses while The Witcher 3 is a valentines special dark magic tray of deluxe bite size wonders with a bottle of sparkling rose! It makes perfect sense to me anyway.

 

[Saltyk]

As much as I appreciate the scores, I rarely think of them as too important. Now, if everyone is giving this game a high score or a low score, that probably means something, but the individual scores mean very little. And I know plenty of games I love get rather middling reviews. Hell, I will sing the praises of Legend of Legaia if you like, and it has a 77% according to Gamerankings. I could probably find other games I love with similarly middle of the road reviews (kinda sucks that 77% is considered middle of the road, though).

There are even plenty of Youtubers who don't give numerical scores. The Completionist simply evaluates if a game is worth completing or not. And Projared often gives rather comical "scores" that are often more informative and nuanced than a simple numerical score.

 

[Jake Martinez]

I don't mind review "scores" depending on how they are used. If they are used by the reviewer to quantify an opinion about something, then that's fine I suppose, if they are used by a system like metacritic completely devoid of context of the original review, then that's garbage because I have no idea why one site rated a game "3/5" and another rated it "4 stars out of 5" or another rated it "83%" without understanding the comments from the review. Because of this lack of context, as a metric Metacritic-like review aggregators mean almost nothing.

Personally instead of stars or numbers or anything like that I prefer just to know if something is like something else. So, when reviewing a FPS, I want to know what other FPS's it's like, what it does better than them (or worse) and about where it sits overall compared to some of the better ones. Just like if I was reading a review of a movie, I expect some sort of understanding about genre to factor into the review. If a movie is a comedy, you'd compare it to other comedies, if it's a specific type of comedy like "road movie" or "stoner comedy" or "sex comedy" then you compare it also to those sub-genres. I hardly ever see game reviewers doing this when it's something that movie reviews do quite frequently.

Finally, because I play mostly on PC, the thing I really care about when buying a game is some idea of the technical specs, like REAL technical specs not just the minimum requirements. I want to know about what I can expect in terms of FPS at a reasonable resolution, how much texture memory would be required for that, etc. I generally end up having to trawl through forums post release and reading what other peoples experiences are with this because it's almost never in a review.

All this being said, I don't hate "scores" but I do hate the way they are used quite frequently. Maybe the happy medium is to just do something like

"If you like games with blah blah and blah, then " buy/don't buy.

Because at the end of the day, that's kind of all we really want to know for a recommendation.

 

[tippy2k2]

You may hate them but I feel that they are a very important part of my buying experience and any sites that don't use scores are sites that I don't use.

It seems like the opposite of what you should do (especially in this internet age) but I want to go into a game as blind as I possibly can. My research into a game is to wait for people to talk about how amazing (or shitty) a game is, find out if it's a game I think I'd like (which is basically seeing the genre and maybe a few descriptors), look at the scores to see if the general population agrees or disagrees, and then buy it.

I know it's probably kind of depressing for a critic to hear that I look at the score and nothing else since I imagine you spent a lot of time writing that (my shitty reviews I do on the side take me at least a half hour to write and I'm nowhere near a professional writer) but all I'm interested in is the score if it's a game that I believe I'd be interested in.

 

[trobert]

It's ironic that the two biggest complaints I hear about video games reviews are the need to eliminate review scores and the need for more objective reviews. I'm not accusing anyone of hypocrisy, I just think it's it's interesting there are these diametrically opposed demands from 2 different camps of gamers.

 

[UberPubert]

I've seen this topic come up now and again and I guess it's an interesting one but I disagree with titling the article this way. Please do better, Escapist.

This is also a little awkward for me, since I don't use reviews as a buyer's guide, I'm just interested in what other people think about a game and why from an observer's point of view. I look at a review score and don't see "huh, so that's how this game is" but rather "oh, so that's what they thought of it". I feel like I never really know if I'll like a game until the end credits are rolling, so trying to discern that from a review or even game play footage feels like a lost cause.

With that said, I don't think it's necessarily a reviewer's task to judge things like style or aesthetic when it comes to the way a game looks or feels, rather that the reviewer should do their best to impart what that style is onto the reader to allow them to make their own decision, instead of including that into a score. There's definitely some art styles and even entire genres of games that I don't like, but if I'm entrusted with giving it a fair and honest review I need to omit my personal preferences on the subject and stick to judging what I'm familiar with. This is where actual experience with video games comes in handy, but unfortunately a lot of modern day reviewers in the video game blogosphere don't have that.

On the subject of scoring systems, I like the stars system so long it's kept to 1-2-3-4-5, any time half-stars get thrown in it makes me think of the 1-10 system, which has been made terrible through its usage on other sites where anything below a 7 is complete trash, which would actually be above a three in a three out of five star review, and I see threes as flawed games with redeemable aspects that could have been executed better and are still worth checking out.

 

[Casual Shinji]

Scores are fine, people just need to stop taking them so goddamn serious, and see them for what they are; a simple numerical representation of the reviewer's overall enjoyment of the game (or movie, or whatever).

- 9 or 10 = great
- 7 or 8 = good
- 5 or 6 = average
- 3 or 4 = below average
- 1 or 2 = bad

It's no different from coming out of a movie and saying 'I loved it', 'I liked it', 'it was okay', or 'I hated it'.

 

[NewClassic_v1legacy]

I like scores. Sorta. Sometimes I don't too.

Like the text to which they're attached, scores mean a lot of different things and all behave differently depending on the context they're put in. Objectively, the number four isn't a lot - the average hand has more fingers - but when talking about the number of cars one owns, the number "four" takes a different context. Four ceases to be a little, and instead becomes a lot. Or, if we're referring to the number of dollars in one's bank account, four isn't much. I've always seen scores that way. They can mean big things, or little things, depending on who and how you ask.

Which is why I find that scores aren't really as bad as the people who misuse them. Those who load up Metacritic or OpenCritic and say "Game averages at 74, so it sucks. Case closed." are the people misusing numbers. They're the sort who read a headline and then hit the comments section. They're the ones who will take half of a concern and turn it into a whole problem. Such an approach is silly to me when recontextualized. Imagine a sign in front of a charity that reads "Want free quarters? Check inside!" Inside are people giving away envelopes with a quarter of a million dollars each. Someone pokes their head in, "How many quarters?" The tellers respond, "Four each." Four quarters of a million. Free quarters. The someone goes, "Pft, that's hardly worth it. Nah." and wanders off.

I mean, granted, this example is extreme, but it's letting someone who makes quick and dirty assumptions about something be the deciding factor on if that something has value. It's such a strange outlook, because scores can say a lot of somethings about an experience, but only if accompanied by context. I'd agree if game reviews were to omit text and exclusively become one-paragraph notes attached to numbered scores, but they aren't. There are still long, wonderful examples of the written criticism attached to these things people terribly misuse. We just only pay attention to the misuse.

I personally use scores when I'm going over a game. If I'm looking into something, I'll start off by popping on OpenCritic, and clicking the highest and lowest scores' reviews, and one somewhere in the middle. I'll read each, make some sense of why the score landed where it did, and between the three pieces, I'll usually gain an understanding of not just what the game was doing, but how well it got there, and if it was worth doing. Having those numbers to give me a quick idea of who to look at if I'm trying to get an abbreviated picture helps me identify more dynamic takes on a game's value, and although the numbers don't mean anything by themselves, they do help me find wider opinions in fewer reviews.

However, I definitely agree that scores are really counter-intuitive sometimes. The Dragon, Cancer is one of the most fascinating games I played this year, it's a wonderful experience that genuinely deserves consideration, contemplation, and I truly believe that everyone should play it. I also scored it a 3 out of 5 [http://www.escapistmagazine.com/articles/view/video-games/editorials/reviews/15259-That-Dragon-Cancer-Review], the lowest score on OpenCritic [http://opencritic.com/game/2126/that-dragon-cancer?tab=1]. And I stand behind that score. As a game, That Dragon, Cancer is jumbled, artful without significant mastery, and sometimes a little bit too inaccessible to really translate as a game everyone should play. But I will fully tell everyone I meet that it's a game everyone should play. It's an amazing experience, and has changed the connotation of "pancakes" forever, but it's certainly not a great game. Well, almost certainly. It's also a great game.

It's a hard thing to pin down.

As they're really understood, scores and aggregate sites aren't communicating the way they're meant to. Review standards mean that a huge divide of things can determine why a score falls the way it does. Taken as a huge picture, scores might tell us a bit more about critics anthropologically. Particularly when we look at the average scores of games like Grand Theft Auto III (97 [http://www.metacritic.com/game/playstation-2/grand-theft-auto-iii]) and Grand Theft Auto - San Andreas (93 [http://www.metacritic.com/game/playstation-2/grand-theft-auto-san-andreas]), and seeing how changes over time can adjust expectations from a game. But at a fixed point in time, just trying to assign one game an average value based on numbered scores form a variety of guidelines, is pretty silly. Scores fail that metric.

I still believe scores do something. They summarize in the same way words grunts and sounds like "Eh" and "Bleh" and "mmhm" do, and whose values are about as oblique without sticking around to see how these vocalizations are rationalized. For me, a score of 4 and 10 are both profoundly useful, but only when accompanied by the texts to which they're attached. Without text, the numbers are meaningless, but with text, the numbers can be meaningful, and that alone gives scores value.

 

[Silentpony_v1legacy]

I feel like this article perfectly shoots the idea of game critics in the foot, while also fully justifying standing its ground because the foot had a knife and was trespassing in the wrong shoe.

Here's the thing, if gaming scores are meaningless, because an opinion is too complex to display in a simple arbitrary 1-10 scale system, then that's fair enough. Even factoring in decimal points, that doesn't leave a lot of room for nuance between an 8.8 and a 9.0.
Having said that, if all game critics are giving their personal subjective opinion, then doesn't that kinda' make the whole idea of a critique...meaningless? I mean the opinion of the game is scene through a lens you, I, Susan Freeman from 4 miles away, Captain Ventris of the Ultramarines and Dracula can't replicate, ie the critic's themselves, assuming of course Uriel hasn't become another damned YouTube Lets-Player.
So then by what right does any critic at all dare to offer their opinion, if their opinion is applicable to themselves and nobody else?

Now to counter my own point, lets say we grant such Daring Do-ness to the critic by trusting them. Fair enough. I think Jim Sterling has a pretty good idea of what makes games good and bad. So I read his reviews and trust him. But then, if you trust them, then why shouldn't they give a numerical score? After all, the critique and score are scene through a lens we've chosen to trust.

So I'd say if we're willing to say 'Critic X's/Aggregate site's opinion of games is valid to me' I think we have to follow that up with 'And X's numerical score/aggregate score attempting to sum up their opinion, however weakly, is equally valid'. I don't think its possible to value an opinion or site while also devaluing a score.

TLR: either you trust the site/reviewer, or you don't. How they choose to publish their opinion is meaningless.

 

[Redlin5_v1legacy]

I don't really care about the numbers. Yahtzee despises them too. I remember when Jim reworked the system here when he was a review editor here but it just never really stuck with me. You generally find out if a game is shit or not the longer you read a review, unless the writer has been drinking the kool aid.

 

[StatusNil]

Damn it, you made me click a Kill Screen link out of curiosity. I'll never get that click back! If I was assigning a score for this article, that would mean an automatic reduction of 1.43 out of Perfect 11.

But yeah, I have to concur with the basic argument. Scores are supposed to be useful on those occasions when you idly browse through discounts and come upon something that you're not familiar with but looks interesting. So you look up the aggregated score to get a rough sense of how well the concept is executed. (After all, it's unreasonable to expect Today's Busy Gamer to devote precious minutes to researching potential purchases.) But what you're really getting with that seemingly authoritative number is a tangle of conflicting agendas that would require much more work to unravel into meaningful coordinates than it is worth. You'd have to know, for an example, that Site X will automatically thrash Dealer Tycoon 4 because of its fundamental ideological opposition to positive portrayals of capitalist exploitation. And Site X doesn't even have the decency to call itself "The Revolutionary Worker's Guide to Decadent Entertainment", but goes by something blandly sleek and abstract, like "Apeirogon".

 

[008Zulu_v1legacy]

I pretty much ignore the number based scores. I do read reviews, you can tell when the reviewer is being honest and when they are pandering. It's not what they write, it's what they don't. While some are pretty good at hiding these unsaid sentiments, if you read between the lines (the specific words, the sentence structure), it's easy to tell the difference between an honest review and a paid review.

 

[AzrealMaximillion]

Scores need to piss off. The defense of scores are also pathetic. "The review is subjective". Then don't put an objective number on it. I hate to say it but Kotaku has done something right. In ridding themselves of score they force a smarter look at their critique of games.

My main critique of scores of that the positives and negative of a game change wildly. Not just by who reviews the game, but even game to game by the same reviewer. Like someone who doesn't like GTA for having bad guys as protagonists but at the same time loves how badass the child soldier using Big Boss is in MGS.

 

[Covarr]

Welp, I guess I need to actually play Mortal Kombat Advance because I can't trust its 33% on Metacritic to mean anything.

P.S. Thanks

P.P.S. In case it isn't clear, I do have an actual point here and not just random snark: While scores may not always correspond to the same scale or have clear factors to make them fair, imperfect is hardly the same thing as useless. As a general rule, good games tend to get higher scores and bad games tend to get lower scores, even if the precise numbers aren't exactly what someone might agree with. Sure, we might not be able to tell a a 7.8 and a 7.9 apart meaningfully, but a 7.8 and a 2.2? Yes, that is a significant and meaningful difference. That 2.2/10 game is almost guaranteed to be utter garbage, and seeing that 2.2 saves you the trouble of even reading the review.

So no, scores are not useless. Just because 6-9 is murky territory doesn't mean the numbers are entirely meaningless. They aren't, and I'm tired of seeing "imperfect" and "worthless" being treated as identical.

 

[CaitSeith]

Scores need to piss off. The defense of scores are also pathetic. "The review is subjective". Then don't put an objective number on it. I hate to say it but Kotaku has done something right. In ridding themselves of score they force a smarter look at their critique of games.

My main critique of scores of that the positives and negative of a game change wildly. Not just by who reviews the game, but even game to game by the same reviewer. Like someone who doesn't like GTA for having bad guys as protagonists but at the same time loves how badass the child soldier using Big Boss is in MGS.

I disagree with the idea that numbers are inherently objective. For that to be true, they must share the same context.

 

[Andy Shandy]

Eh, scored reviews are fine. My only complaint with scores (on the reviewer side, at least) is that 1-5 is barely ever used. Going by most websites' own scales that means pretty much every game is somehow above average. Jim Sterling and Videogamer are the only two I've seen that tend to use the whole scale.

Oh, and publishers should not be using them - particularly Metacritic's weighted scores - to dole out bonuses.

 

[Tilly]

I'm sticking with scores Lizzie!

I need a quick way to check a lot of different opinions. I don't have time to read through more than 1 or 2 full text reviews. Well what if those 2 reviews are heavily anomalous. People who just don't like the genre or had a bad day, or whatever else.
I need a vague average to get a quick idea about what most people think. Aggregate scores are that way!

 

[Baresark]

Oh man... did I just read this right? A reviewer who admits their reviews are opinions... You deserve some sort of award for that alone. I can't even tell you the number of times people have tried to argue reviews are "objective".

OT: It's true, the number thing is a terrible way the industry has gone. I'm not saying that they need to be gone, they are useful in the way twitter is useful. If you want to say something with absolutely no context what so ever, use a numbers score, but ALWAYS view it as a small part of a much larger thing. Sometimes, when I wonder about a game, I'll read a handful of full reviews, maybe watch a youtube video about it, but after that I get the general feelings of the game and the score is more useful as a reminder. This is especially true of the entire games media community feels basically the same about a game. As an example I would say the Uncharted games fit this.

Scores become utterly useless when the games media community cannot agree on whether a game is good or bad. At that point, scores should be ignored or people should be encouraged to read whole reviews.

 

[bdeamon]

Scores are fine, people just need to stop taking them so goddamn serious, and see them for what they are; a simple numerical representation of the reviewer's overall enjoyment of the game (or movie, or whatever).

- 9 or 10 = great
- 7 or 8 = good
- 5 or 6 = average
- 3 or 4 = below average
- 1 or 2 = bad

It's no different from coming out of a movie and saying 'I loved it', 'I liked it', 'it was okay', or 'I hated it'.

That makes sense to someone who reads a lot of reviews, but what is the baseline of "average" for someone who does not play a lot of games, or has no reference of best and worst parts of a genre (racing games for example). It is not like review sites have guides on how to read their scores, and how they are different to other site's ratings. I mostly agree with how you placed the numbers, but from what I see, IGN considers 7 to be average, and Polygon places its average at 5, and that's not even mentioning the quality of the written review can be very different as well.