A Review Scoring System That Would Work

MrHide-Patten

New member
Jun 10, 2009
1,309
0
0
oldtaku said:
The purpose of the review score is to tell us whether to bother to read the review for a game we weren't interested in.

9-10: Wow, guess I should read the review to figure out what's so great about this. Maybe I'm interested now.
1-3: Ooo, guess I should read the review to enjoy the train wreck.
4-8: I'm not reading the cunting review.

Review sites could achieve the same effect by going with better titles or ledes. '[Game]: The [Site] Review' doesn't really do it.
Usually a good review can paste out the tone or quality of a game from the first couple of lines. Ergo Jim Sterling's Review of Dying Light; 'Thumbs Down At Sundown, A promising game, brought low by its own lack of ambition.'
A neat to the point summery of banality.

There's too much meaning attached to numbers to the point that if a Nintendo game gets any lower then a 7 the whole internet erupts into wailing dipshits.
 

coheedswicked

New member
Mar 28, 2010
142
0
0
This is why I like your video reviews, Yahtzee. I don't have to read anything and you pick everything apart quite neatly ( and harshly).
 

The Rogue Wolf

Stealthy Carnivore
Legacy
Nov 25, 2007
16,302
8,778
118
Stalking the Digital Tundra
Gender
✅
Thunderous Cacophony said:
I wonder where something like Sunset Overdrive would fit on that scale? Breaking the 4th wall is the definition of losing immersion, but if done well it can be enjoyable and shouldn't be punished.
I think that's where context comes in. It's when the game otherwise tries to present itself as a serious and immersive experience but fouls it up that it should be docked.
 

klaynexas3

My shoes hurt
Dec 30, 2009
1,525
0
0
See, I'll have to side with Yahtzee on this one, both in that you should read a whole review, and that breaking apart the scoring system would be a good thing, because there's far more to games than what they once were. They are not just a simple product, they are far more intricate now, so objectively rating a game becomes not possible due to how many layers exist within these games. Take a game like Fallout New Vegas. Plenty of bugs and brokenness in this game, and yet the roleplaying is great, the combat has depth, there's multiple different ways to play the game, etc. How would we objectively review this game? Do we look at its bugs and say it's a 5/10, do we look at the play style, the gameplay itself, is it great for being deep, is it bad for being too complicated, if it's good for providing more options, how does that fair with a game like Mario, a tight platformer, but a rather linear and straightforward game, would it lose out due to it's lack of depth, how would this all work?

See, everyone speaks about having an objective review, and yet there is no way to boil a game down into one numerical score objectively. You have to make assumptions, biases, opinions, all varying depending on the game, because each game should be reviewed for what it is alone, and due to the vast amount of different games that exist, there is no way to boil it down into one simple system to rate games. So, the only solution is to break things up, understand certain aspects have to be taken with general bias, or drop the system entirely, and maybe go with something like Eurogamer's "recommended, essential, whatever" system, and include what those badges would correspond to just so it's a nice summary to know if the review is worth looking into, and drop metacritic as well. The problem with aggregated scores is that trying to base your own taste on general consensus is a horrible idea, and to be an educated consumer, you have to be willing to go in depth on what you're buying. The critics say Final Fantasy XIII is a good game. Objectively speaking, it works just fine. Ask any Final Fantasy fan however, and they'll tell you why it's shit, and even provide a laundry list of valid, objective reasons as to why that is. So the only way to even be close to having an idea of if what you are about to buy will be good is to find a reviewer, or sets of reviewers that align with your tastes. That's the smart way to go about being a consumer.
 

Not Lord Atkin

I'm dead inside.
Oct 25, 2008
648
0
0
I tend to be wary when people start calling for complete abolishment of certain things, even if there is nothing inherently wrong with the thing they are trying to get rid of. It's been happening a lot in professional circles lately. Like when Totalbiscuit recently went on about day 1 DLC never ever being justified (and pretty much starting a rant about on-disc DLC without distinguishing the two, forgetting about how large chunks of development teams tend to be sitting around twiddling their thumbs between their game going gold and its actual release date); Shamus hoping to ban 2-weapon limits (despite them just being a tool in a designer's box that fits certain games and doesn't others) and... well... this.

Look, the fact that a thing is being misused does not mean it's an inherently bad thing. You wouldn't ban the use of kitchen knives because most people can't cook anyway and some use them for amateur lobotomy.
Don't start a revolt against knives just because you don't like curry and everyone who chops their chicken into bits is obviously not in it for the meat and is therefore stupid and just fuck knives, they are not objective enough being a part of this inherently subjective genre of food preparation and this analogy is starting to break down a bit. I'm very tired, what am I still doing on the internet.

I'll just shut up.
 

Geisterkarle

New member
Dec 27, 2010
282
0
0
Yahtzee, you are lying! You gave a game a score! It was Wolfenstein and you awarded it 2 out of 5 stars!
So your article is DESTROYED!!
 

blackrave

New member
Mar 7, 2012
2,020
0
0
I think linear scoring does not give justice to most games
Yahtzee is on to something
How about this
Each stat is rated 0-10

Graphics(&Aesthetics)
Performance(optimization, bugs, etc.)
Sound(quality of music and voices)
Story(originality, presentation, ending)
Gameplay
Fun (overall feelings after playing this game)

And since it is internet review we could add exact reasons behind decreased score (upon clicking on stat)

For example

Shadow of Shadow: Dawn of Reawakening
Graphics 7
-2 very low quality textures
-1 generic character design

Performance 5
-1.5 very limited configuration options
-1 game is unable to use multiple GPUs
-1 game is capped at 30FPS
-1.5 several bugs

Sound 6
-2 highly repetitive (and eventually annoying) music
-2 unprofessional (and sometimes even bad) voice actors

Story 4
+1 main villain is amazing
-1 game starts with amnesia
-2 failure to provide any interesting plot element
-2 story breaks internal lore multiple times without providing any explanation
-1 ending is banal and "plot twist" can be predicted in the middle of story
-1 setting is banal and done better several times in other games

Gameplay 8
-1 AI could use a bit more polish
-1 GUI is clunky

Fun 7
-1 somethingsomething
-1 blablablablablabla
-1 wordswordswords

I'm not sure how easy/hard would be analyzing games like this, but I would like to watch at least few attempts of it.
 

beerit

New member
Feb 18, 2015
1
0
0
There's a solid scientific basis to subjective numerical scoring systems, what Daniel Kahneman referred to as "Intensity Matching" in his book "Thinking Fast and Slow".

From the book:
"An underlying scale of intensity allows matching across diverse dimensions.
If crimes were colors, murder would be a deeper shade of red than theft.
If crimes were expressed as music, mass murder would be played fortissimo while accumulating unpaid parking tickets would be a faint pianissimo.
And of course you have similar feelings about the intensity of punishments. [...] If you heard two notes, one for the crime and one for the punishment, you would feel a sense of injustice if one tone was much louder than the other".

I see no reason not to intensity match my experiences with numbers (among other things), or allow myself to feel emphatic enough to read another person's numerical intensity matching of a video game and draw my own conclusions.

On a strictly personal note, the scores I give games in Metacritic are almost always close to the cumulative user scores (critic scores are almost always off, for obvious reasons).

Why? Because the human brain kicks ass at intensity matching.

So why limit the tools at our disposal, when we have this ability just waiting to be used with the calculative speed and force of an infinitely creative supercomputer?
 

Kahani

New member
May 25, 2011
927
0
0
BrotherRool said:
I don't see the problem with review scores. They're just very rough approximations of how positive the reviewer feels toward whatever they're reviewing.

It's only ever an issue when people expect the system to be perfect.
Exactly. Review scores are a great way to give you an idea of whether it's even worth bothering to read the review in the first place. If a game is getting solid 2s and 3s out of 10, I can be pretty sure it's shit and not worth my time investigating any further. If it's a game I've been looking forward to and am fairly sure I want to buy, getting 9 or 10/10 is a good indication it's not a broken piece of crap and I'll probably enjoy it without needing to read any further. Scores in the mid range indicate that some people will enjoy it but reading some reviews in more detail would be a good idea to figure out if I'm one of them.

Scores are only a problem if people only look and the raw score and never go any further than that. All criticism of scores that I've seen rest entirely on the assumption that that's exactly what the vast majority of people do, but I've never seen any evidence that this is actually the case. Sure, you get lots of internet shouting about how a game was given the wrong score by some reviewer, but there's no indication that anyone saw Call of Battlefield 12 only got a 9 and was forced not to buy it.

As for score inflation, there is simply no such thing and the complaints about it make no sense whatsoever. Yes, a score of 7/10 is seen as fairly mediocre. That's exactly what it's supposed to mean. There's absolutely no reason a score of 5/10 should be considered normal, any more than a score of 50% is always the pass mark for exams. The sort of scale reviews actually work off looks something along these lines:

1-2 - broken and unplayable.
3-4 - playable, but that's the best that can be said about it.
5-6 - the game works and isn't entirely terrible, but has serious flaws that mean it won't be attractive to most people.
7 - a decent game, potentially enjoyable to those who like whatever sort of game it is.
8-9 - really quite good, may be worth looking at even if you're not usually into that sort of thing.
10 - fucking awesome.

The bottom end of the scale is rarely seen not because of some imaginary inflation, but simply because no competent developer should ever produce a product that doesn't actually work. Any game produced by paid professionals and funded by a publisher should be hitting around 5 at the absolute minimum, because anything lower means that they have completely failed at their job. Obviously indie games will be seen more in the bottom end because they tend to be made by small groups without the same financial and technical support.

Equally important is that most games at the bottom end of the scale don't get reviewed. There's tons of utter shit around that probably deserves such scores, but for the most part neither consumers nor reviewers care about it in the slightest. Professional book reviewers don't tend to give scores to the tons of self-published shit on Amazon either.

Combine the two effects and it should be extremely obvious why review scores are usually at least 6 or higher. No competent developer should produce anything that scores less, and no competent reviewer will bother looking at games that are likely to score lower. 7 is considered a relatively poor score because it essentially means you've achieved the bare minimum; developers don't like that because no-one likes hearing "eh, it was OK I guess" about the thing they've spent at least a couple of years working their hardest on, and publishers don't like it because it means their millions invested in it aren't likely to result in huge profits and a lucrative franchise for the future. That doesn't mean there's anything wrong with review scores, it just reflects the fact that when there are thousand of products to choose from, consumers, developers and publishers all only want the very best, and "good enough" is not the very best.
 

Scow2

New member
Aug 3, 2009
801
0
0
I like review score numbers. They do serve a point beyond 'people who don't want to read'... and most review scores I find are worthless only if you don't read the review. Here on the Escapist, with just one review a week or so, the numbers at the end of a review give a solid reminder of how good a game is, cutting through and clarifying the overall impression where just the text could give a different impression, such as if it meanders and gushes over a great/potentially great mechanic or moment in a bad game, or nitpicks a flub in a great game.
 

Biran53

New member
Apr 21, 2013
64
0
0
Consumers should be informed and intelligent.

This arises not from scavenging Metacritic for the best game scores, but reading/listening to the critiques from a preferred critic and making a smart purchasing decision based on your own tastes or intuition. Fixating on the numbers and GOTY is a pointless endeavor, and this consumer behavior does not lead to better games. I suppose we don't have to discard numeric criticism entirely, but so many examples have proven how broken a system it is time and time again.
 

Comrade_Beric

Jacobin
May 10, 2010
396
0
0
"I've never given a score since day one" You famously gave the Wolfenstein reboot two stars in your limerick review.

SEE, SOMEBODY LIKED THAT DAMNED EPISODE. I'm still in shock that it got knocked out in the first round of the community choice ZP reviews. Now, back to reading your article...
 

ryukage_sama

New member
Mar 12, 2009
508
0
0
It'd be nice to hear from multiple reviewers on each site. I know coming to Zero Punctuation who is going to review a game, but I rarely get that reliability from other sites. If my tastes perfectly lined up with Yahtzee, I might not need to read many reviews to gauge if a game is worth my time and money. This might be the reason so many YouTube personalities have gained a following. Their viewers know that every review on the channel is going to be by the same person, time-after-time.
 

gsilver

Regular Member
Apr 21, 2010
381
4
13
Country
USA
One thing that I like about Yahtzee's scoring system:

Something like GTA V or Skyrim would be "0% functional" games, because the larger and more complicated a game is, the more bugs it's going to have.

Meanwhile, a cowclicker could easily get a 100% under the same system, since with very little interaction and simple mechanics, it's much easier to make it be bug-free.
 

Ishal

New member
Oct 30, 2012
1,177
0
0
Thanatos2k said:
That said, if video games wanted to go with a rottentomatoes approach with an aggregated pass/fail to produce a quality score I'd be right behind that.
https://www.indiegogo.com/projects/basedgamer-com

Then get behind it.

OT: Data is good, but seeing it via video is better. Reviews are nice, and they are useful when gauging if I should spend my money.

I enjoy long form videos. I'll watch a 20-30 minute video if I think it will give me useful information. Same with a long written review if no video is available. I can multitask, and I am a boring person.

shrekfan246 said:
Zhukov said:
shrekfan246 said:
I rather firmly believe that the problem is the readers. Yahtzee touched on it a little, but it's the people who come into a review just to reaffirm their preconceived notions; when they see a number they disagree with for whatever reason, they have to throw a massive tantrum all over the internet because apparently other people holding different opinions is just something they cannot reconcile with their world view.
This is pretty much why everyone reads reviews though.

Hell, we see people getting mad over reviews that fail to confirm their feelings on games they haven't even played yet.

The only time people read reviews for actual information or opinions is when the review is of a game they haven't heard of and they're reading it out of curiosity.
I can't speak for other people, but I read reviews to get information even if it's a game I've been following.

More often I watch videos of the game in question, admittedly, as there are few reviewers or critics I follow who regularly put out written content, but I even read Jim Sterling's review of Final Fantasy X HD before I bought it, despite it being a re-release of a game I've owned for over a decade.

Honestly, sometimes I really do feel like I'm some kind of robot when it comes to the area of review and critique in video games, because I just can't fathom how incensed people get over these subjects. It makes no sense to me.
That's because you have this miraculous outlook on life. Where you don't attach your personal worth and identity to your interests, specifically in the games you buy.

Great, isn't it? I think you'll find a lot of people have this outlook. Certainly a lot more than the people Zhukov is referring to.
 

Olrod

New member
Feb 11, 2010
861
0
0
I like scores. I read all the lowest scoring reviews of popular games and think to myself "Could I put up with these problems?"

If I can regard the only things that reviewers have a problem with as "not a big deal" then I'm probably going to enjoy the game.
 

Thanatos2k

New member
Aug 12, 2013
820
0
0
Ishal said:
Thanatos2k said:
That said, if video games wanted to go with a rottentomatoes approach with an aggregated pass/fail to produce a quality score I'd be right behind that.
https://www.indiegogo.com/projects/basedgamer-com

Then get behind it.
Campaign's long closed. We'll see what they come out with.

OT: Data is good, but seeing it via video is better. Reviews are nice, and they are useful when gauging if I should spend my money.

I enjoy long form videos. I'll watch a 20-30 minute video if I think it will give me useful information. Same with a long written review if no video is available. I can multitask, and I am a boring person.
30 minute videos? That's a half an hour I could be using to play a game though.
 

Thanatos2k

New member
Aug 12, 2013
820
0
0
StreamerDarkly said:
I find this article to be full of terrible logic that attempts to support a preconceived conclusion, which, ironically, is something the author complains about in attempting to explain why review scores are bad.

The worst flaw is probably the idea that aggregate statistics, such as an average score, "merely flatten things out and become meaningless". To address this problem, it is claimed that we really need to see every single data point from some 20 or 25 independent reviewers to derive any meaning. Clearly not the case, as data sets of any size can often be well characterized by just two parameters: mean and variance of the sample.

Actually, the initial claim that 'averages are useless' because things get 'flattened out' is the sort of cringeworthy insight I'd expect from a 2nd year general arts student. Averages are great in certain types of analysis, particularly in the case of video games where the little niggle that would cause one player to dock marks is completely irrelevant to 90% of the gaming population. More precisely stated, every observation (individual review score) in the data set is contaminated by noise (personal preference that doesn't transfer to the majority of other players), and the only way to extract what we might call the signal (more objective assessment of the game's quality) is to compute the average rating. Obviously not everyone will agree that the average score reflects the game's quality, but it is the single number that MOST players will agree with.

To think you can just say "art is subjective" and be done with it is the height of ignorance.
The real reason reviewers try to argue that "averages are useless" and "averages flatten out stuff" is that they're afraid of what averages really do - render them just a voice in a sea of other reviews, all equal.

They don't want to be equal though. They want their review to mean more than everyone else's. They think they know better after all, and they want you to read THEIR review (and please click through to our site while you're at it).

When I can see 50 reviews at a glance, their scores, and the average of their scores, I only need to fully read a few reviews to get the gist of why the scores fell where they did. This is poison to reviewer ego.
 

Fireprufe15

New member
Nov 10, 2011
177
0
0
BrotherRool said:
I don't see the problem with review scores. They're just very rough approximations of how positive the reviewer feels toward whatever they're reviewing.

It's only ever an issue when people expect the system to be perfect. I can give a book a 4 out of 5 because I think it's really good but not the most amazing thing ever. My thought out opinion is always going to be more useful, but 4/5 is a nice way to put those thoughts in context.
Review scores would be less of a problem if Metacritic didn't exist. Because now these scores based on different systems by different people who feel very different about a subjective piece of art need to be compiled into one score that is supposed to give you a summed up idea of what the general public thinks of a game. And that doesn't work. Ever.
 

BrotherRool

New member
Oct 31, 2008
3,834
0
0
Fireprufe15 said:
BrotherRool said:
I don't see the problem with review scores. They're just very rough approximations of how positive the reviewer feels toward whatever they're reviewing.

It's only ever an issue when people expect the system to be perfect. I can give a book a 4 out of 5 because I think it's really good but not the most amazing thing ever. My thought out opinion is always going to be more useful, but 4/5 is a nice way to put those thoughts in context.
Review scores would be less of a problem if Metacritic didn't exist. Because now these scores based on different systems by different people who feel very different about a subjective piece of art need to be compiled into one score that is supposed to give you a summed up idea of what the general public thinks of a game. And that doesn't work. Ever.
But that's also only a problem if you expect it to be a perfect system. Metacritic is great if you just accept that its a rough tool to be used with intelligence.

Take this usage:

"Lets go see a film. What shall I watch?

Well I've never heard of Seventh Son, and it's got a 31 rating on metacritic, so lets not bother with that. But this Selma film has got a 81, I wonder what that's about? The Kingsman has got a 59, but I really liked the look of it, so I'll check that out to.

[click on Selma and Kingsman]

Oh I see most Kingsman has a load of really good reviews and then a couple of really bad ones pulling the average down. So what's good about it? [clicks a few positive reviews] And what do people hate? [clicks a few negative reviews]

Huh, I wasn't expecting it to be so violent. Colin Firth is cool and all but I'm not sure if I really want to watch a film where they kill a bunch of stand-ins for real-life civilians even if those people are really horrible.

Lets see what the reviews of Selma say..."


That process is 10x more time consuming and annoying without review scores and metacritic. Without metacritic you can't instantly filter out Seventh Son, you have no idea where to find the good reviews and the bad reviews and nothing highlights Selma as something you can watch. And without review scores you can't deliberately seek out reviews with contrasting opinions to find out what the good/bad points are.