In defense of the number: a note on video game review scores

[Mutant1988]

I don't understand how you don't get what a numerical value offers. It's not like people (rational people) what a number score just to see how high there favorite game scores or so they don't have to read a review because they are lazy.

Firstly, a numerical score is useful because many times the text of a review will be more positive or negative than the reviewer wanted. Maybe they spent more text on the game's flaws (which is fine) or they felt there really wasn't much wrong with the game but just didn't dig it that much.

Here's the thing - A review that does not reflect the actual views of the writer is a poor review. The addition of a number at the end doesn't change that. If the text makes the game come across as a terrible game, yet the reviewer likes it, then that reviewer is a very poor writer.

Furthermore, if you write a negative review, but still end up recommending the game, that means that to the writer the positives he or she did bring up weigh more than all the negatives. That's all it takes to make a seemingly negative review a positive one, with reservations.

There's a lot of things bad with this game, but the things that are good make up for it - In the opinion of the writer. That's an important thing to keep in mind - It's an opinion. It can be an informed opinion, but it's still an opinion.

Secondly, it's nice to see how someone feels about one game vs another (in the same genre) to help you decide which one to buy now and what to try later if they both came out during the same time.

And I argue that you shouldn't rely on how people "feel" at all. I read reviews to find descriptions of features that I enjoy, or believe that I would enjoy. That and to find out if there is any element of a game that makes it a non-starter for me (Excessive QTE, poor lock-on aiming systems).

I very rarely make a poor purchase, because I know specific elements to seek out and avoid. That and I look at the aesthetic and thematic qualities of a game first, with the consideration for gameplay being primarily - Does it work at all and would I hate it?

Thus why I read multiple positive and negative reviews. The more text to elaborate on what the game actually does (Right and wrong), the better.

Just reading written text of both reviewers isn't going to be that helpful in trying to figure out which game the reviewer thought was better. Also, it's nice to have a quick way to see what a reviewer thought about a series of games (how he/she ranks all the Metal Gears, Halos, CODs, GTAs, etc.) so you can determine if they like the same aspects that you like about series, which is very informative when they go to review the latest sequel coming out.

I don't get this obsession with "better". The only part that matters is - Is this game good for me or not? If Game Y is better than game X, but I'm reading the review for game Y, then obviously it's more interesting to me.

I don't play a rating. I play a game, with gameplay mechanics and themes that I enjoy. The numerical ratings distract from the assessment of those things. Or at the very least, contribute absolutely nothing to it.

Other mediums use number scores too and the professional criticism within those mediums is just fine. I don't think anyone is saying to score a game, a number exactly represents the reviewer's feelings on a game either. However, that number is more informative than not having a number. Just because you can use something the wrong way doesn't mean it doesn't have real legitimate uses.

The numbers in reviews of other mediums is equally arbitrary. If you base your media consumption solely on the basis of what is best, then you ignore the specific qualities of the product.

It's an over reliance on popular perception (What everyone else likes) rather than learning what you like and how to look for it.

An aggregate of Yes and No would be pretty much just as useful as the current Metacritic 100 point scale aggregate. Which is to say, not very good at all for actually judging the quality of a specific product. A high aggregate tells you that a lot of reviewers like it, but not why (Or why not). So you would still end up with majority voted bad games at the bottom, and good games at the top - But avoid the obsession with numbers entirely.

I fail to see how a simple Yes/No tally would be less useful, when you still need to read actual reviews to learn about the product.

 

[Mutant1988]

He wasn't. Fappy's messed with his account to make it look like he's banned whilst still retaining full posting privileges.

How delightfully arbitrary and not confusing at all.

 

[Phoenixmgs_v1legacy]

Here's the thing - A review that does not reflect the actual views of the writer is a poor review. The addition of a number at the end doesn't change that. If the text makes the game come across as a terrible game, yet the reviewer likes it, then that reviewer is a very poor writer.

Furthermore, if you write a negative review, but still end up recommending the game, that means that to the writer the positives he or she did bring up weigh more than all the negatives. That's all it takes to make a seemingly negative review a positive one, with reservations.

There's a lot of things bad with this game, but the things that are good make up for it - In the opinion of the writer. That's an important thing to keep in mind - It's an opinion. It can be an informed opinion, but it's still an opinion.

Every reader will interpret things differently. Every reader will read the review and judge how positive or negative it was and that will vary. The number helps make sure you and your readers are on the same page.

Reviewers are about criticizing so the review should come off as more negative IMO. I think that's very much lacking in game reviews, there's very little actual criticism going on. Every game sounds like it's good.

And I argue that you shouldn't rely on how people "feel" at all. I read reviews to find descriptions of features that I enjoy, or believe that I would enjoy. That and to find out if there is any element of a game that makes it a non-starter for me (Excessive QTE, poor lock-on aiming systems).

I very rarely make a poor purchase, because I know specific elements to seek out and avoid. That and I look at the aesthetic and thematic qualities of a game first, with the consideration for gameplay being primarily - Does it work at all and would I hate it?

Thus why I read multiple positive and negative reviews. The more text to elaborate on what the game actually does (Right and wrong), the better.

I think a lot of what makes someone love a game has quite a few intangibles to it. I want a reviewer's feeling on a game. There's plenty of games that I play where there's nothing really wrong with the game but you just don't dig it for whatever reason. I really dug Binary Domain even though the gameplay does have quite a few issues as I ended up really enjoying the story and characters. Whereas I found Max Payne 3 to be possibly one of the worst games I ever played, not because it was broken or unplayable or had features I didn't like; I hated the writing and the gameplay just had a bunch of small issues (from shooting mechanics to level design) that added up that made me hate the gameplay as well.

I don't get this obsession with "better". The only part that matters is - Is this game good for me or not? If Game Y is better than game X, but I'm reading the review for game Y, then obviously it's more interesting to me.

I don't play a rating. I play a game, with gameplay mechanics and themes that I enjoy. The numerical ratings distract from the assessment of those things. Or at the very least, contribute absolutely nothing to it.

There's so many good games, good movies, good TV shows, etc. You only have limited time to take in all this content. I would like to experience all good things but there isn't the time for that. I'd rather spend my time on the things I'd really enjoy vs the things I'd mildly enjoy.

The numbers in reviews of other mediums is equally arbitrary. If you base your media consumption solely on the basis of what is best, then you ignore the specific qualities of the product.

It's an over reliance on popular perception (What everyone else likes) rather than learning what you like and how to look for it.

An aggregate of Yes and No would be pretty much just as useful as the current Metacritic 100 point scale aggregate. Which is to say, not very good at all for actually judging the quality of a specific product. A high aggregate tells you that a lot of reviewers like it, but not why (Or why not). So you would still end up with majority voted bad games at the bottom, and good games at the top - But avoid the obsession with numbers entirely.

I fail to see how a simple Yes/No tally would be less useful, when you still need to read actual reviews to learn about the product.

The whole point of professional critics is that it's not popular opinion. Rarely, if ever, does the highest grossing movie win Best Picture. Usually the GOTY is one of the highest selling games, which is very unlike all other mediums. I play/watch whatever I think will be good for myself, I don't rely on anyone to tell me what experience. But it is nice to find a niche thing you love, then find a critic that loved that same thing and then look over what else he/she loved that you either didn't know about or didn't check out and then end up finding new stuff that you really dig just due to finding a critic with similar tastes. Without giving things a numeric rating, you can't really do that kind of stuff outside of top ten lists.

I very rarely use aggregate scores at all. I look up aggregate scores more in threads like this to prove game reviewers are fucked up than I do when actually deciding what game to buy. An aggregate good/bad like RottenTomatoes is more useful IMO than an aggregate score is.

Lastly, I think your dislike for aggregate scores is more the reason for you hating reviews being scored more than anything. If you don't want a number, just read the review and don't even look at the number as it's at the end of the review anyways. Adding a number merely gives people more options and if you want the option of not having numbers, you can already do that with or without numbers. You want to enforce your beliefs on everyone else when you can already have want you want. There's a lot more wrong with game reviews than numbers, game criticism is kinda non-existent at this point and only a few reviewers actually properly criticize games like critics of other mediums. I find game reviews completely pointless and meaningless at the state they are in now.

 

[babinro]

Numbers are a must for me.

I'm a visual person and time is valuable. When I'm looking at reviews for a game that I'm considering I just need a quick glance to get the idea. With this initial data I can quickly determine of more in depth research is needed.

The point where game scores no longer matter to me is when I've decided I WANT to buy the game but then research to learn its flaws and find out what might turn me away from the purchase.

In other words...a review score acts as a massive time saving tool when the game that's peaked your interest is unknown to you. As is the case with many of the humblebundle sales. I'm not about to take hours reading several reviews from multiple sites on 10-15 games.

 

[Mutant1988]

Every reader will interpret things differently. Every reader will read the review and judge how positive or negative it was and that will vary. The number helps make sure you and your readers are on the same page.

I disagree. If you interpret the text differently from how it's written, you're not on the same page, score or no score.

Even then a simple yes or no recommendation would have the same effect, without the headache of trying to figure out what a 7 rating for Gameplay is supposed to mean.

Reviewers are about criticizing so the review should come off as more negative IMO. I think that's very much lacking in game reviews, there's very little actual criticism going on. Every game sounds like it's good.

This I can agree with. I try to bring up any aspect of a game that I can imagine being an issue, or were an issue for me. Because I find it's easier to like things that are less than perfect, but impossible to play a game when specific elements bring the entire experience down (Like QTEs or bad lock on).

I think a lot of what makes someone love a game has quite a few intangibles to it. I want a reviewer's feeling on a game. There's plenty of games that I play where there's nothing really wrong with the game but you just don't dig it for whatever reason. I really dug Binary Domain even though the gameplay does have quite a few issues as I ended up really enjoying the story and characters. Whereas I found Max Payne 3 to be possibly one of the worst games I ever played, not because it was broken or unplayable or had features I didn't like; I hated the writing and the gameplay just had a bunch of small issues (from shooting mechanics to level design) that added up that made me hate the gameplay as well.

I don't really see how that in any way relates to the numerical rating. To know whether this game does things you won't like, you need to read the text. The text, essentially, is the feelings of the reviewer. The number is just an abstract summary that doesn't mean anything on it's own.

Case in point, Max Payne 3 reviewed excellently. But as you say, there's a ton of minor issues with it that could not in any way be communicated in a number.

There's so many good games, good movies, good TV shows, etc. You only have limited time to take in all this content. I would like to experience all good things but there isn't the time for that. I'd rather spend my time on the things I'd really enjoy vs the things I'd mildly enjoy.

Honestly, you want to know my main method for selection? Tvtropes. Find a trope I like, check what works it apply to - Voila, entertainment purchases planned. Of course, I still read reviews to find out whether games have something I just can't stand. I also keep an eye on recent releases (On this site, for example), in case there's something that sticks out in my mind.

The whole point of professional critics is that it's not popular opinion. Rarely, if ever, does the highest grossing movie win Best Picture. Usually the GOTY is one of the highest selling games, which is very unlike all other mediums. I play/watch whatever I think will be good for myself, I don't rely on anyone to tell me what experience. But it is nice to find a niche thing you love, then find a critic that loved that same thing and then look over what else he/she loved that you either didn't know about or didn't check out and then end up finding new stuff that you really dig just due to finding a critic with similar tastes. Without giving things a numeric rating, you can't really do that kind of stuff outside of top ten lists.

It is a matter of popular opinion when people look at aggregates. It's just that it relies on the popular opinion of a majority that is assumed to know better. As for the Oscar and similar awards... It's easy to argue that quality has nothing to do with the selection process. Rather, it's whether any given work fits into a pre-determined measurement of "Oscar Quality". And it also depends on whether the maker or actor is due for a reward.

I do think that Top 10 lists and such as far superior. Personal preference, a quick summary of the work, a nice and easy overview of games a specific reviewer liked for you to browse and pick out games that look interesting.

But yeah, I don't see how a list with Yes/No recommendations from any given reviewer wouldn't work just as well as a list of numerical scores from the same. Add a genre filter to it and you can find recommendations or any kind of game you would like to know about.

Lastly, I think your dislike for aggregate scores is more the reason for you hating reviews being scored more than anything. If you don't want a number, just read the review and don't even look at the number as it's at the end of the review anyways. Adding a number merely gives people more options and if you want the option of not having numbers, you can already do that with or without numbers. You want to enforce your beliefs on everyone else when you can already have want you want. There's a lot more wrong with game reviews than numbers, game criticism is kinda non-existent at this point and only a few reviewers actually properly criticize games like critics of other mediums. I find game reviews completely pointless and meaningless at the state they are in now.

No, I sincerely believe that people that skip to the end of a review to see the score and then move on to the next site of reviews is missing the point of a review. It is to convey information. Specific information. The numbers are not specific enough.

I really like Yahtzee because he examines the game and shares his experience, rather than just run it through a checklist to produce a score.

I think more than anything else, the obligation to include a score dumbs down media critique more than anything else. As well as the ability of customers to know whether the product they purchase is good for them specifically or not.

A simple "This is good" or "This is bad" with a text explaining why is all a critique should contain. If it doesn't even reach the minimum standards of it's medium or genre, it shouldn't be rated below 5 (Or 4 - After which it's essentially nuances of shittiness). It should be stated plainly in the review - This shit doesn't work.

One of my most favourite game series is the Earth Defence Force series. If I were to review that by some universal notion of quality then I'd be forced to give them a low score for their many technical failings. But I'm having too much fun to care about the erratic frame rate, laughable physics and spotty texture work.

How is that represented in numbers, that one element far, far outweighs the others? I wouldn't even put a technical rating for such a game. I'd just stick a disclaimer right at the top of the review - Don't buy this game if you cannot stand erratic frame rates, laughable physics, spotty texture work and terrible (But awesome) voice acting.

Then the entire rest of the review would be me telling the reader how it's just fun to play. Fun, fun, fun! Not quantifiable by any mathematic formula.

Though possibly measurable by magnetic brain scans. Hmm... Maybe we should hook every reviewer up to a brain scanning machine to determine if their enjoyment is genuine or not.

Or we stop trying to turn critique of entertainment into a science and just learn what we like or do not like and how to sort through information to determine if a product is or isn't to our liking.

 

[Phoenixmgs_v1legacy]

If you interpret the text differently from how it's written, you're not on the same page, score or no score.

Even then a simple yes or no recommendation would have the same effect, without the headache of trying to figure out what a 7 rating for Gameplay is supposed to mean.

Tone isn't very well conveyed through writing. Even with numbers, it's still a yes/no recommendation, having a number just adds to what degree the yes/no is. Like Siskel & Ebert giving thumbs up/down, they still reviewed the movies based on the 4-star system.

7 means above average, how is that hard to understand?

Case in point, Max Payne 3 reviewed excellently. But as you say, there's a ton of minor issues with it that could not in any way be communicated in a number.

Right now reviewers won't mark down games they dislike if the game is functional and all. They won't even heavily score down a game if they disliked the story in a story-based game (like FFXIII). I would rate Max Payne 3 as like a 2-3/10 because I hated the game, I thought it sucked regardless if the shooting was competent or even good. The only game reviewer that I even know that will score games based on solely on how much or how little they liked the game is Jim Sterling. With movies, you'd have Ebert saying this movie sucked and Siskel saying it was one of the best of the year. That doesn't have with game reviews because reviewers try to objectively rate games instead of just rating them based on how much they liked the game.

It is a matter of popular opinion when people look at aggregates. It's just that it relies on the popular opinion of a majority that is assumed to know better. As for the Oscar and similar awards... It's easy to argue that quality has nothing to do with the selection process. Rather, it's whether any given work fits into a pre-determined measurement of "Oscar Quality". And it also depends on whether the maker or actor is due for a reward.

I do think that Top 10 lists and such as far superior. Personal preference, a quick summary of the work, a nice and easy overview of games a specific reviewer liked for you to browse and pick out games that look interesting.

But yeah, I don't see how a list with Yes/No recommendations from any given reviewer wouldn't work just as well as a list of numerical scores from the same. Add a genre filter to it and you can find recommendations or any kind of game you would like to know about.

Again, just because scoring games allows for aggregates doesn't make scoring games inherently bad. You can kill someone with a knife, that doesn't mean knives shouldn't be allowed because they have their uses. The yes/no recommendation is part of the score; over 5 = recommended, under 5 = not recommended. 7 has become the new 5, which is a problem that game reviewers caused themselves. Nobody else scores art where a 7 equals average.

Aggregating yes/no's for games would be far worse. Almost every game would be like 99% buy, 1% don't buy. Even a game like Destiny only has 1 negative review.

No, I sincerely believe that people that skip to the end of a review to see the score and then move on to the next site of reviews is missing the point of a review. It is to convey information. Specific information. The numbers are not specific enough.

That doesn't affect you though. So what if other skip to the score or just merely go to Metacritic. No one is forcing YOU to do that.

I really like Yahtzee because he examines the game and shares his experience, rather than just run it through a checklist to produce a score.

I think more than anything else, the obligation to include a score dumbs down media critique more than anything else. As well as the ability of customers to know whether the product they purchase is good for them specifically or not.

A simple "This is good" or "This is bad" with a text explaining why is all a critique should contain. If it doesn't even reach the minimum standards of it's medium or genre, it shouldn't be rated below 5 (Or 4 - After which it's essentially nuances of shittiness). It should be stated, plainly, in the review - This shit doesn't work.

I like Yahtzee but his reviews aren't really reviews and many people confuse his videos for him hating every game. His Uncharted 2 review is very negative but he listed it as one of the better games that year. I've learned that whenever he hates on a game for non-gameplay aspects, he probably liked it overall. That's not a good review system at all.

All other mediums don't have an issue with scores dumbing down the critique.

Games have gotten to the point where they almost always are function. 3rd-person shooting has gotten really good for example; before PS3/360, 3rd-person shooting was so poor it took a survival horror game (RE4) to get the genre to a decent standard. The Order 1886 has very competent shooting, does that automatically make the game good? It's nothing special for a TPS to have good shooting nowadays to the point where I don't care if the shooting is good (that is the new average), I care if it's done really well or uniquely at this point, and that is rather subjective now. We've gotten to the point in gaming that making a functional game isn't special anymore. I don't even know if gamers, on the whole, really care much about bugs/glitches (early access games and such are a different story) as I'm sure Bethesda's next game will be pre-ordered in the millions and will have lots of bugs/glitches.

 

[Mutant1988]

Tone isn't very well conveyed through writing. Even with numbers, it's still a yes/no recommendation, having a number just adds to what degree the yes/no is. Like Siskel & Ebert giving thumbs up/down, they still reviewed the movies based on the 4-star system.

I've never recommended anything as so-so. I tell people what is good about it and it's up to them to look that up or take my word for it, if they are interested in playing the game. If I tell someone that a game is great or awful and nothing else, I'm not doing them any favours whatsoever. Why is it good? Why is it bad? That's the important bit.

7 means above average, how is that hard to understand?

Above average what? The technical standard? Then I'm sorry, but you can just drop everything below 7 then. There's also the whole idea of what constitutes average for any one person.

This game is average does not tell anyone anything about the game. Not a single detail. If there was a universal standard for average, then it might have some meaning. But there isn't and there will never be. Does the game run? Okay, then it's entirely possible that someone can like it as much as a 10/10 (Because it's just that fun) or detract from it enough to put it below 5/10 (Because it's just that boring).

Adhering to some universal scale doesn't do anyone any favours. If anything, it distracts from the point of a critique, by enabling people to skip the actual critique itself.

That doesn't affect you though. So what if other skip to the score or just merely go to Metacritic. No one is forcing YOU to do that.

It does affect me that other people aren't being smart customers. Throwing good money at games you don't like does no one any favours. Well, except for the publishers. They love getting your money for any reason. You being happy with the product usually doesn't matter.

I like Yahtzee but his reviews aren't really reviews and many people confuse his videos for him hating every game. His Uncharted 2 review is very negative but he listed it as one of the better games that year. I've learned that whenever he hates on a game for non-gameplay aspects, he probably liked it overall. That's not a good review system at all.

He reviews games. The fact that he doesn't cover every aspect of it doesn't make it any less of a review. There is a very good reason why I keep repeating the advice to read multiple reviews. Because it really is impossible for any one person to cover absolutely everything, good and bad.

All other mediums don't have an issue with scores dumbing down the critique.

In weighing technical merit versus creative merit, as interpreted by the reviewer? They most certainly have. A movie with a stupid plot but excellent cinematography? Oh, can't have that below average. Excellent story but awful pacing? That's got to be a terrible movie then. Extremely entertaining but completely mindless and shallow? Well, people will like it so got to rate it high.

That is, if we are supposed to judge consensually determined (Not "Objective") technical merit along with the subjective creative merit. If we just rate it by if it's good or not, there's really no need for a scale. It's a simple yes or no.

Same with books. How do we rate different types of prose? Is a first person narrative more immersive than a third person? Or is the important part whether we like it or not and the reasons we state for why? Yes, that's the important and only important part to a critique.

The reasons for why we like or dislike a work is the only important part of a critique. You cannot summarize the sum of quality of different elements where the appreciation of each element also differ from person to person.

Games have gotten to the point where they almost always are function. 3rd-person shooting has gotten really good for example; before PS3/360, 3rd-person shooting was so poor it took a survival horror game (RE4) to get the genre to a decent standard. The Order 1886 has very competent shooting, does that automatically make the game good? It's nothing special for a TPS to have good shooting nowadays to the point where I don't care if the shooting is good (that is the new average), I care if it's done really well or uniquely at this point, and that is rather subjective now. We've gotten to the point in gaming that making a functional game isn't special anymore. I don't even know if gamers, on the whole, really care much about bugs/glitches (early access games and such are a different story) as I'm sure Bethesda's next game will be pre-ordered in the millions and will have lots of bugs/glitches.

I don't even know what point you're making here. So we should stick with a numerical rating because it's absolutely pointless to have one? I mean, everything would be 7/10 based on technical merit. Or should we remove that and only rate fun? Then maybe 3/10 for the competent but completely lifeless and uninspired? No, that doesn't make sense. So got to have the technical merits in the numbers too, but by which ratio? 50/50?

Screw it. It's just distracting from the entire point of a review. Which is to tell you why exactly this game is good or bad in the opinion of the reviewer.

Sorry if I'm repeating myself but what I'm saying just seems to incredibly obvious to me, it's just baffling that you guys don't "get" it. That's not an insult or anything, I'm just not seeing your point at all.

I don't really understand what is supposed to be "good" about a numerical review. That it exists so people can ignore the actual review? How is that a good thing? That's a terrible way to decide what games to buy.

Mind, I think top lists work just fine. Why? Because it gives context to the selection. It still won't tell you that my pick for game 1 is better for you than game 10, but it will tell you that I like those games and if you have similar tastes... Well, might be a good idea to check out the other games on the list.

 

[Phoenixmgs_v1legacy]

Above average what? The technical standard? Then I'm sorry, but you can just drop everything below 7 then. There's also the whole idea of what constitutes average for any one person.

This game is average does not tell anyone anything about the game. Not a single detail. If there was a universal standard for average, then it might have some meaning. But there isn't and there will never be. Does the game run? Okay, then it's entirely possible that someone can like it as much as a 10/10 (Because it's just that fun) or detract from it enough to put it below 5/10 (Because it's just that boring).

Adhering to some universal scale doesn't do anyone any favours. If anything, it distracts from the point of a critique, by enabling people to skip the actual critique itself.

Buy it or don't buy it has the same issue, there's no universal standard because everyone's different. Some people won't buy a game if it isn't X amount of hours long regardless on its quality.

It does affect me that other people aren't being smart customers. Throwing good money at games you don't like does no one any favours. Well, except for the publishers. They love getting your money for any reason. You being happy with the product usually doesn't matter.

No matter what you do there's always going to be lots of people you feel that aren't smart consumers. It still doesn't affect you at all. So what if John Johnson buys a game he doesn't like. And, no system will guarantee he buys a game he likes either.

He reviews games. The fact that he doesn't cover every aspect of it doesn't make it any less of a review. There is a very good reason why I keep repeating the advice to read multiple reviews. Because it really is impossible for any one person to cover absolutely everything, good and bad.

Zero Punctuation aren't real reviews. His Extra Punctuation articles are way more informative than his reviews, I'm sure less people read those than watch Zero Punctuation.

I don't even know what point you're making here. So we should stick with a numerical rating because it's absolutely pointless to have one? I mean, everything would be 7/10 based on technical merit. Or should we remove that and only rate fun? Then maybe 3/10 for the competent but completely lifeless and uninspired? No, that doesn't make sense. So got to have the technical merits in the numbers too, but by which ratio? 50/50?

Screw it. It's just distracting from the entire point of a review. Which is to tell you why exactly this game is good or bad in the opinion of the reviewer.

Sorry if I'm repeating myself but what I'm saying just seems to incredibly obvious to me, it's just baffling that you guys don't "get" it. That's not an insult or anything, I'm just not seeing your point at all.

I don't really understand what is supposed to be "good" about a numerical review. That it exists so people can ignore the actual review? How is that a good thing? That's a terrible way to decide what games to buy.

Mind, I think top lists work just fine. Why? Because it gives context to the selection. It still won't tell you that my pick for game 1 is better for you than game 10, but it will tell you that I like those games and if you have similar tastes... Well, might be a good idea to check out the other games on the list.

Numbers CAN be useful. I like Jim Sterling as a reviewer. It's nice to be able to see what games he gave 9-10 scores to for example. You'll see Lollipop Chainsaw and Majin and The Forsaken Kingdom on that list. Those games didn't score that on Metacritic for example. That's not possible without numeric scores. I'd rather play something someone passionately liked than something someone just liked.

 

[StreamerDarkly]

[HEADING=3]The Diversity Hire: A Tale of Jamal Sterling[/HEADING]

Everyone here knows the name of Jim Fucking Sterling, son. They'll also be familiar with his recent decision to strike out as an independent games reviewer, thus far a successful endeavour to the tune of $10K per month on Patreon. It's a much more uplifting story than that of Bob 'MovieBob' Chipman, who, despite also being a content producer of some acclaim, has only managed to secure $3.6K per month, scarcely enough to pay the bills and, rather depressingly, even less than certain Twitterinas [footnote]The Twitter equivalent of a Tumblrina[/footnote] who ostensibly work in the games industry but produce nothing of value.

A more interesting question, however, is what Jim Sterling has been up to lately. In the context of this thread and in order to advance the discussion of selection bias among professional game critics, let's take a look at the fruits of The Jimquisition's labor over the past four months.

Now, you should understand that Jim Sterling isn't a man who has ever been afraid to drop the hammer on a game he doesn't like. He's actually somewhat acclaimed for this trait. Not to the extreme that Yahtzee Croshaw takes it, mind you, but nevertheless a stern taskmaster when it comes to the appraisal of video game quality.

Without any further preamble, let's have a look at The Jimquisition's review scores since November of last year. The histogram below portrays the typical clustering of scores in the 50-100 range. The average score is 70.6. To be fair, it appears that Jim was sitting on a backlog of high quality titles he wanted to share his thoughts on (Grand Theft Auto V and The Binding of Isaac: Rebirth, for example), which would naturally inflate his average rating a tad. In my opinion, the real point of interest is at the low end of the scale.

An argument can easily be made for the relevance of Sonic Boom; it's Sonic the fucking Hedgehog after all. Even Dynasty Warriors managed to draw considerable attention from critics [footnote]It's rare that a game accumulates more critic reviews than user reviews on Metacritic. In general, this occurrence does not speak highly of the game's quality.[/footnote] despite its low standing. But what is this Haunted House: Cryptic Graves game on about? Just two other critics took an interest in this title, both rating it at 10/100 as The Jimquisition did. Sadly, or perhaps fortunately in the present case, three reviews isn't even enough to secure a proper Metascore.

If you haven't guessed it by now, I'm pointing out how trivially easy it is for a games reviewer to manipulate their average score by choosing from a semi-infinite population of lesser known titles of poor quality. We might never know why Jim Sterling chose to review this particular one, it really doesn't matter.

Imagine that you're a games reviewer who prefers, for the sake of appearances, to be near 70 on the average. After all, if you let things slide up towards 80, that's GameTrailers and Game Informer territory which no self-respecting reviewer wants to touch with an arsenic-coated barge pole. It's trivially easy to achieve this target by picking an occasional sacrificial lamb to be dumped on. For example, if you decide to review 1 horrendous game out of every 20 reviewed, this can pull an average of 75 down to

75*(19/20) + 10*(1/20) = 71.8

where the horrendous game has been assumed a rating of 10/100. A game rated at 20/100 or 30/100 does the job nearly as well.

Naturally, some will argue that there can't possibly be anything wrong with the simple act of reviewing a bad game and giving it the score it deserves. I agree. It's simply a matter of how many critics do (or don't do) this on a regular basis for the sake of massaging appearances, and whether the audience is mindful of it. Realize that randomly stomping on a no-name indie game does nothing to change the scores already awarded (and that will continue to be awarded) to AAA and medium budget titles. The disparity in game selection by reviewers could therefore by very important when judging how 'honest' their ratings are.

In conclusion, the practice concept of 'managing statistics' by games reviewers can be loosely analogized to the following conversation between the head of HR and the CEO of a fictitious company at the staff Christmas party:

Linda: "Great party. Full turnout. The buffet is bountiful, but I can't help but notice that the sausages walking about easily outnumber those laid out on the table."
John: "Hmm, what are you saying?"
Linda: "If there's a blizzard in town, it has much less to do with the weather outside or the stimulants available in the bathroom than the complexion of employees we're current looking at.
John: "I see what you did there."
Linda: "We need to increase diversity, pronto!"
John "Do what you must, Linda."

 

[freaper]

Why would a (indie) reviewer criticize games his/her audience is never going to want to play? Would Jim F. Sterling want to review the next MLP:FiM game, knowing that maybe two guys watching his review would be interested in playing that game? It's the trinity of consumer-hype-reviewer; hype is created for a product, the consumer wants to know more about said product and the journalist provides a review which in turn will generate more hype. It's only logical for a journalist to want to cover the "good" stories. Reporting on ISIS seems a lot more prestigious and lucrative than writing an article on Timmy getting hit by a car.

On a personal side note; if you're going to write an academical thesis, leave out the bile, it's not flattering.

 

[StreamerDarkly]

Why would a (indie) reviewer criticize games his/her audience is never going to want to play? Would Jim F. Sterling want to review the next MLP:FiM game, knowing that maybe two guys watching his review would be interested in playing that game?

For the sake of credibility, as I described. You don't think someone like Jim Sterling who has talked about scores as many times as he has is cognizant of what an inflated average score looks like? It isn't necessarily some sort of deliberate strategy, it's just one reason you might see a difference in aggregated statistics between reviewers who (more or less) operate with the same reviewing standards.

On a personal side note; if you're going to write an academical thesis, leave out the bile, it's not flattering.

This isn't an academic journal, it's an internet discussion board. I don't care to flatter anyone. If you have anything relevant to say about the content of the article, I'll be happy to listen.

 

[StreamerDarkly]

Numbers not necessary.

In fact, I'd argue that the Steam Review format is one of the best out there. A simple "Do I recommend this Yes/No" rating system and as much text as you need to explain why you would or wouldn't.

Especially useful to read several recommendations as well as non-recommendations, as it gives a more complete picture of the advantages and disadvantages of the product. The latter being especially important for PC games, which aren't guaranteed to work at all.

OK, but what Steam also does is take those recommendations from users and compile them into an overall approval rating of the form:
1. Overwhelmingly positive (95%-100%)
2. Very positive (80%-95%)
3. Mostly Positive (70%-80%)
4. Mixed (40%-70%)
5. Mostly negative (25%-40%)
6. Very Negative (15%-25%)
7. Overwhelmingly negative (0%-15%)

I'd argue that these ratings are quite helpful to users of Steam. You can see immediately what should be avoided, what's top quality, and for the middle categories read a few of the review blurbs to make up your mind. It's vastly more efficient than reading through hundreds of individual comments, a lot of them repeating the same things, to get a feel for how good a title is.

Seven rating categories is actually quite a lot of precision, too. It's fundamentally the same information that Metacritic provides in its user section, with the caveat that on Metacritic you're allowed to say to what degree you liked or disliked a game, and the ratings are binned into only 3 categories (Positive, Mixed, Negative).

One problem I see with the Steam system is, because of their frequent sales, they end mixing reviews from players who paid a different price for the game. Price can change your expectations of a game and overall impression significantly. On the other hand, at least you can be sure the player bought the game unlike on Metacritic.

 

[Scow2]

Lastly, 5/10 is fucking average, not 7/10.

No. 7/10 is average, just like in grade school. 5/10 is terrible, just like in Grade School.

 

[Mutant1988]

OK, but what Steam also does is take those recommendations from users and compile them into an overall approval rating of the form:
1. Overwhelmingly positive (95%-100%)
2. Very positive (80%-95%)
3. Mostly Positive (70%-80%)
4. Mixed (40%-70%)
5. Mostly negative (25%-40%)
6. Very Negative (15%-25%)
7. Overwhelmingly negative (0%-15%)

I'd argue that these ratings are quite helpful to users of Steam. You can see immediately what should be avoided, what's top quality, and for the middle categories read a few of the review blurbs to make up your mind. It's vastly more efficient than reading through hundreds of individual comments, a lot of them repeating the same things, to get a feel for how good a title is.

Seven rating categories is actually quite a lot of precision, too. It's fundamentally the same information that Metacritic provides in its user section, with the caveat that on Metacritic you're allowed to say to what degree you liked or disliked a game, and the ratings are binned into only 3 categories (Positive, Mixed, Negative).

One problem I see with the Steam system is, because of their frequent sales, they end mixing reviews from players who paid a different price for the game. Price can change your expectations of a game and overall impression significantly. On the other hand, at least you can be sure the player bought the game unlike on Metacritic.

That rating is based on the sum of two parameters. Like or dislike. The individual reviews do not have any kind of percentile rating, because it's one opinion from one person. The point of it is to indicate how many like or dislike the product. Not to equate the exact degree of quality of the product.

If I were against counting that I would be against democracy. Because that's essentially what that is.

And it is not fundamentally the same information as metacritic provides, where arbitrary metrics are applied to the numbers each user gives. The fact that numbers mean different things and are reached by different means, depending on who awards it, makes the numbers useless.

If you want to know if a game is good or not, read the reviews. Efficiency? I thought the point was to find fun games, not to weed out undesirables. But maybe I'm just too open minded about games.

 

[StreamerDarkly]

And it is not fundamentally the same information as metacritic provides, where arbitrary metrics are applied to the numbers each user gives. The fact that numbers mean different things and are reached by different means, depending on who awards it, makes the numbers useless.

Bah, this argument of vastly different scales for different reviewers is overblown. The similarity in results from the various critics proves it.

A better question might be: if the Steam and Metacritic systems lead to the exact same results once enough votes are tallied, how can you say that one system is useless just because you have a technical quibble with it?

Here is an alternative interpretation of review scores. Forget about the detailed lists of what each number means just for a second. Instead, a score can be viewed as a prediction of the likelihood that a player would enjoy the game. So, if The Jimqusition gives a 7/10 rating and it ends up that 70% of players on Steam also recommend it, that's an excellent prediction indeed.

 

[Mutant1988]

And it is not fundamentally the same information as metacritic provides, where arbitrary metrics are applied to the numbers each user gives. The fact that numbers mean different things and are reached by different means, depending on who awards it, makes the numbers useless.

Bah, this argument of vastly different scales for different reviewers is overblown. The similarity in results from the various critics proves it.

A better question might be: if the Steam and Metacritic systems lead to the exact same results once enough votes are tallied, how can you say that one system is useless just because you have a technical quibble with it?

Here is an alternative interpretation of review scores. Forget about the detailed lists of what each number means just for a second. Instead, a score can be viewed as a prediction of the likelihood that a player would enjoy the game. So, if The Jimqusition gives a 7/10 rating and it ends up that 70% of players on Steam also recommend it, that's an excellent prediction indeed.

There is a stark difference in the sum of opinion of all people and the opinion of a single one. Especially in the method by which they are reached and/or measured.

The former is just counting how many said like or dislike.

The latter is one person quantifying their sum "like or dislike" of the game, based on factors that are judged differently from reviewer to reviewer and not accounting for factors that could make or break the game for any specific user (Only sure fire way to find those out is to read positive and negative reviews).

There is no scientific worth in a (Singular) rating. Nor is there in tallying such ratings. Tallying how many say Yes, No or Maybe?

That's actually a far better indicator for how likely it is that someone would like a thing. In fact, that gives you the straight up numbers you seek, with zero room for interpretation. But as a metric of some sort of universal quality? No such thing.

 

[StreamerDarkly]

There is a stark difference in the sum of opinion of all people and the opinion of a single one. Especially in the method by which they are reached and/or measured.

The former is just counting how many said like or dislike.

The latter is one person quantifying their sum "like or dislike" of the game, based on factors that are judged differently from reviewer to reviewer and not accounting for factors that could make or break the game for any specific user (Only sure fire way to find those out is to read positive and negative reviews).

There is no scientific worth in a (Singular) rating. Nor is there in tallying such ratings. Tallying how many say Yes, No or Maybe?

That's actually a far better indicator for how likely it is that someone would like a thing. In fact, that gives you the straight up numbers you seek, with zero room for interpretation. But as a metric of some sort of universal quality? No such thing.

You leave out a very critical point. If numbers are problematic because everyone has a different measuring scale, then so too are likes/dislikes problematic for the same reason.

Distilling a very analog opinion of a game which resides in your brain down to yes/no doesn't get around the issue at all. Because not only do the analog opinions differ significantly among game players (which you can tell from review comments), but so too do the threshold levels of individuals which the analog result will be compared to to compute like/dislike.

Not everyone has the same tolerance for what they would call bad and good. More clearly, over a very large collection of games, the ratio of likes to dislikes will vary between players. Possibly by a lot. The individuals will not meet the illusory 50% average any more than a professional critic will.

So why should I consider like/dislike some sort of sacred paradigm of true meaning?

 

[Mutant1988]

You leave out a very critical point. If numbers are problematic because everyone has a different measuring scale, then so too are likes/dislikes problematic for the same reason.

If the purpose was to determine if a lot of people like the game, then a simple yes/no tally is perfectly sufficient.

It is still not a substitute to actually reading a review. But it alone is as good a metric as the arbitrary scales when it comes to deciding whether to bother or not.

You yourself even said so. If it produces the same results with less ambiguity, why not use it instead? Why even bother with the 1 to 10 scale when counting the sum of all opinions produce the same percentage ratio?

Trying to quantify a single opinion on a scale based on the sum of the merits of other elements at some kind of fixed ratio causes more problems than it solves. It's adding complexity to where it's not needed.

Distilling a very analog opinion of a game which resides in your brain down to yes/no doesn't get around the issue at all. Because not only do the analog opinions differ significantly among game players (which you can tell from review comments), but so too do the threshold levels of individuals which the analog result will be compared to to compute like/dislike.

Would I recommend this game? It's a simple yes and no answer.

If there's a but, it needs to be followed by a reason. A number is not a reason. It's devoid of meaning and arbitrarily chosen to determine what is "better" or "worse". Which is just nonsense to me, since there's no such thing as objective good or bad - Only a general consensus (Which is still subjective) of what is bad.

If anything, I'd recommend everything "but-" followed by reasons for why someone might love or hate the game. A number will simply not convey that information.

So why should I consider like/dislike some sort of sacred paradigm of true meaning?

You shouldn't.

I don't believe I've ever said such a thing.

It's better than the arbitrary ratings that are common practise. But it's still not a good metric for whether "you" specifically will like a certain game.

Knowing what you like or dislike and looking through the available information to find those things is how you do that.

This is a bit of a recurring theme in my posts but I do insist that people "bother" looking at things and critically examine them instead of relying on faux-scientific ratings.

Read the reviews of games that interest you, for any reason. If you want to find the best of the best, then the games with the most reviews are probably where to start - But actually accurate in describing the popularity (Not quality) of the product.

 

[Phoenixmgs_v1legacy]

Lastly, 5/10 is fucking average, not 7/10.

No. 7/10 is average, just like in grade school. 5/10 is terrible, just like in Grade School.

No other medium is rated where 7 is average because it's stupid.