Lords of Shadow 2 Studio Boss: "One Must Be Blind or Stupid" to Give It a 4/10
- Thread starter Alex Co
- Start date
Still wrong. 5.5 isn't the median and it isn't the average/mean. It's simply the middle of the scale.Steve2911 said:
I certainly do hope you bother to look it up because if you said that in a Elementary math class, Stats class, or even basic algebra you're going to fail. You are fundamentally wrong at a very basic mathematical concept.
If electrical appliances were expected to be 'reviewed' the same way games are, you might see scenarios like this unfolding:
Consumer: This toaster is functional enough to heat a single piece of bread, but the residual heat from toasting makes cooking successive pieces impossible to manage, as they all come out burnt. The crumb tray is also very loose and the electrical contacts are poorly soldered, as you can often see flashes of electricity when you press a button. All in all, I'd rate it 4 out of 10.
Toaster manufacturer: You must be blind or stupid. Look how pretty the toaster is! It's at least worth a 7/10.
Consumer: This toaster is functional enough to heat a single piece of bread, but the residual heat from toasting makes cooking successive pieces impossible to manage, as they all come out burnt. The crumb tray is also very loose and the electrical contacts are poorly soldered, as you can often see flashes of electricity when you press a button. All in all, I'd rate it 4 out of 10.
Toaster manufacturer: You must be blind or stupid. Look how pretty the toaster is! It's at least worth a 7/10.
Average/Mean and Median are defined by the set. Not the scale.Steve2911 said:
In your set of {1,2,3,4,5,6,7,8,9,10} your average is 5.5 as well as your median.
However, in the set {1,5,5,5,5,10,10,10,10,10,10,10}
The median is 10
The average is 7.5
The min is 1
The max is 10
The Scale could be 1 to 10, but it could also have been a scale of 1 to 100. The Average is always defined by the set. Not the scale.
Another common misconception is that half fall below average. That only happens when the set is Normally distributed. In the set I gave only about 40 percent fall below average.
The Average Score given to a video game is about 7.
That's not relevent. The score given is the opinion of the reviewer, not of the reviewing media as a whole. There are many sensible reviewers who would score a game they consider to be average with a 5 or a 6. What other reviewers do is none of their concern.medv4380 said:
It's really not a difficult concept to grasp.
Again completely wrong. In that case Average would be the average of their scores. Their opinion is irrelevant to the average score they give. Someone can "say" that 5 is average, but when you average out all their scores for all games and it comes out to 7 the average is 7.Steve2911 said:
Pick 10 games and 2 review outlets and I'll show you how wrong you are.
To all the people who are saying "5 is average, I don't get the complaints"...most video game reviewers, for better and for worse, use 7 as the average, probably because most video game reviewers are American and have been schooled that 7/10=70%=C=average. You can say all you want that 5 out of 10 should be average, but that's not how video game reviewers work.
Since 7 is average, a game getting 4 out of 10 is pretty bad, and I can understand frustration when you make something and it is slammed by a critic because he didn't like it. This is hardly the best way to voice that frustration, though.
By the way, I use a 5/10 is average scale when I review stuff. However, I make sure to explain that I am using that scale, because to people used to 7/10 being average, me giving 6/10 to something I found flawed but slightly above average would seem contradictory. I'm not sure if Edge bothers to explain that 4/10 to them might mean slightly below average.
Since 7 is average, a game getting 4 out of 10 is pretty bad, and I can understand frustration when you make something and it is slammed by a critic because he didn't like it. This is hardly the best way to voice that frustration, though.
By the way, I use a 5/10 is average scale when I review stuff. However, I make sure to explain that I am using that scale, because to people used to 7/10 being average, me giving 6/10 to something I found flawed but slightly above average would seem contradictory. I'm not sure if Edge bothers to explain that 4/10 to them might mean slightly below average.
So...the general public doesn't trust reviewers. Now game developers don't trust reviewers. Man, that must be a hard job to enjoy. Personally it sounds like this guy got his feelings hurt that his game isn't wowing people.
Alright, I've read the comments and the article and I think I know what I want to say.
Eric Alvarez is totally right about the lack of professionalism when it comes to gaming reviews. I'm going to ignore LoS2's party in what I have to say because I haven't played the game and am not foolish enough to act as if I can tell the games quality without having done so. Nevertheless he's right.
Gaming reviews for the most part never speak about gaming mechanics or quality of things like the UI. Instead what we get is one of two things. One is a very vague summary of the game with limited information on how the game plays, if its buggy, if the UI is functional and displays info in an efficient manner, or how long the reviewer played the game. An example of this (and yes I'm firing shots here) the Escapist reviews of Dragon Age 2 and Total War Rome 2.
Both games were praised and given high score. In the case of Dragon Age 2 there was no mention that the game's options were severely limited compared to the first game (City world instead of country, no race choices for protagonist, and literal copy pasted maps). The review failed to inform people of any of those changes to the franchise and that'd kind of integral to how a review works. There was also no mention on how the mechanics of the game worked other than a slight mentioning of the streamlined combat. I think we all know the general consensus of DA2 isn't a 5/5. There was an Editorial, also on the Escapist, aptly titled "When Dragon Age II Fell Apart". Give it a read.
With the Total War Rome II review the review was just a vague description of the first few hours of the game (a say "few" hours but this is a strategy game so "few" really means anything under 20 hours) and failing to inform the readers that the game was fundamentally broken and not fit for sale in its state at the time of release. I don't know how a game that straight up didn't work got a 4/5 when it objectively did not work as it should have, and that's the problem I have with a lot of reviews these days.
The other caveat I have with "professional" reviews is the reviewer going on a soapbox against a certain aspect of the game and completely ignoring the rest of the game's feature. Then scoring the game based on the one aspect. The Escapist GTA5 review comes to mind with this(I'm sorry I keep going after the Escapist here but this is our community and I want to see it become aware of these problems). A lot of us here on the forums disagreed with the review, not because of the score, but more because the review was more of a rant than a review. The reviewer went on about how he didn't enjoy playing an actual morally corrupt person in a world that depicts everyone as kind of dickish in GTA5. Which really is like complaining about playing a lumberjack in Lumberjack Simulator 2014. Every GTA game had a bad guy protagonist, yet it wasn't OK when the game's story was written as more of a serious satire. Now that's the reviewers opinion and I respect that and choose to disagree, but to not mention any of the features of the game, how the game mechanically plays, how well the game ran, how well its side missions fit into the game or how the UI works out. It was just a soapbox against playing the bad guy in a video game set in a realistic dark satirical America. People in the forums brought up how you could levy the same criticisms against Breaking Bad, but at the end of the day Breaking Bad is very well acted and written and GTA5 was a damn great functioning game. Not liking something doesn't equal it not being objectively functional.
This is why numbered review systems are a problem. If there's no mention of objective facets of a game like how much it lags or if the gameplay is awkward to control, then you're really just putting a number beside a subjective opinion and trying to smoosh them together. And sometimes that subjective opinion isn't even well informed. I'm suspect of any review written in the Oct-Dec rush of games because reviewers go into overload trying to play as many games as possible in as short a time as possible. You throw something like a Total War or Crusader Kings game in there and you will guarantee a poorly informed review due to the time it takes to learn those kinds of games.
Eric Alvarez is totally right about the lack of professionalism when it comes to gaming reviews. I'm going to ignore LoS2's party in what I have to say because I haven't played the game and am not foolish enough to act as if I can tell the games quality without having done so. Nevertheless he's right.
Gaming reviews for the most part never speak about gaming mechanics or quality of things like the UI. Instead what we get is one of two things. One is a very vague summary of the game with limited information on how the game plays, if its buggy, if the UI is functional and displays info in an efficient manner, or how long the reviewer played the game. An example of this (and yes I'm firing shots here) the Escapist reviews of Dragon Age 2 and Total War Rome 2.
Both games were praised and given high score. In the case of Dragon Age 2 there was no mention that the game's options were severely limited compared to the first game (City world instead of country, no race choices for protagonist, and literal copy pasted maps). The review failed to inform people of any of those changes to the franchise and that'd kind of integral to how a review works. There was also no mention on how the mechanics of the game worked other than a slight mentioning of the streamlined combat. I think we all know the general consensus of DA2 isn't a 5/5. There was an Editorial, also on the Escapist, aptly titled "When Dragon Age II Fell Apart". Give it a read.
With the Total War Rome II review the review was just a vague description of the first few hours of the game (a say "few" hours but this is a strategy game so "few" really means anything under 20 hours) and failing to inform the readers that the game was fundamentally broken and not fit for sale in its state at the time of release. I don't know how a game that straight up didn't work got a 4/5 when it objectively did not work as it should have, and that's the problem I have with a lot of reviews these days.
The other caveat I have with "professional" reviews is the reviewer going on a soapbox against a certain aspect of the game and completely ignoring the rest of the game's feature. Then scoring the game based on the one aspect. The Escapist GTA5 review comes to mind with this(I'm sorry I keep going after the Escapist here but this is our community and I want to see it become aware of these problems). A lot of us here on the forums disagreed with the review, not because of the score, but more because the review was more of a rant than a review. The reviewer went on about how he didn't enjoy playing an actual morally corrupt person in a world that depicts everyone as kind of dickish in GTA5. Which really is like complaining about playing a lumberjack in Lumberjack Simulator 2014. Every GTA game had a bad guy protagonist, yet it wasn't OK when the game's story was written as more of a serious satire. Now that's the reviewers opinion and I respect that and choose to disagree, but to not mention any of the features of the game, how the game mechanically plays, how well the game ran, how well its side missions fit into the game or how the UI works out. It was just a soapbox against playing the bad guy in a video game set in a realistic dark satirical America. People in the forums brought up how you could levy the same criticisms against Breaking Bad, but at the end of the day Breaking Bad is very well acted and written and GTA5 was a damn great functioning game. Not liking something doesn't equal it not being objectively functional.
This is why numbered review systems are a problem. If there's no mention of objective facets of a game like how much it lags or if the gameplay is awkward to control, then you're really just putting a number beside a subjective opinion and trying to smoosh them together. And sometimes that subjective opinion isn't even well informed. I'm suspect of any review written in the Oct-Dec rush of games because reviewers go into overload trying to play as many games as possible in as short a time as possible. You throw something like a Total War or Crusader Kings game in there and you will guarantee a poorly informed review due to the time it takes to learn those kinds of games.
Skimming metcritic let's see
The lowest is 33 rating, which spends most its time harping on the stealth segments. Including several outright false statements about them. The stealth segments making up something like 5% or less of it. Oddly, the part of their review that isn't ranting at the minor part of the game is fairly abundant with praise for the new combat.
Edge starts out in similar vein. Goes on to complain about the voice acting, giving Patrick Stewart a pass just cause, then skewering everyone else without much detail. Goes on to call the city London, which it definately isn't if you actuallly paid any attention. Goes on to complain about the climbing being highlighted and set paths, cause apparently they wanted Assassin's Creed where you can just grab everything. They start in on textures and texture pops, none of which I've ever seen, and even on PS3 (the PC looks better), haven't noticed. Maybe they're stuck on next-gen mode, when this is still last gen. They sort of half-praise the combat, then critique it somewhat validly. Then go right off the rails with erroneous statements about how its impossible to stun enemies with block counters or any attacks. They close out with some opinions on the story which are valid enough.
Having read that, I can empathize with the guys opinion. They blatantly get several not-obscure facts wrong about the game. They open by it calling it clunky, but then later complement the core mechanic (combat), outside of apparently being terrible at it or not paying any attention in the tutorial by what they got wrong, and then cited as faults. Even with rampant errors, I don't get a "subpar" out of their article overall as written.
Not to be unobjective myself here, the Toronto Sun (of all places), actually (after some brief ranting about how this Dracula is not their Dracula), covers its critical points fairly well, without peppering misinformation throughout. Reading less like someone with a grudge against the rebooting (or Konami itself) scrabbling for justification, and more like someone who legitimately disliked the game.
To borrow some phrasing from a webshow I watched recently, I think part of the problem is there's a growing crop of reviewers who are also dabbling (or throwing themselves wholeheartedly at) also being entertainers with distinct personas. The ranting cynic being an overly popular one. Its creating a growing distortion and negative slant on the whole process. Yahtzee, AVGN, or the Spoony Experiment, and similar affairs are some flagships of the "Comedic sketch of a review" format, and it seems to be seeping into regular reviewers. We're getting more reviews that seem like they're looking to establish their entertainment persona, with less focus on the actual reviewing behind it. This is highlighted when entire articles go into rant mode on one segment of a game, and ditch the entire other 70-90% into a single sentence.
The lowest is 33 rating, which spends most its time harping on the stealth segments. Including several outright false statements about them. The stealth segments making up something like 5% or less of it. Oddly, the part of their review that isn't ranting at the minor part of the game is fairly abundant with praise for the new combat.
Edge starts out in similar vein. Goes on to complain about the voice acting, giving Patrick Stewart a pass just cause, then skewering everyone else without much detail. Goes on to call the city London, which it definately isn't if you actuallly paid any attention. Goes on to complain about the climbing being highlighted and set paths, cause apparently they wanted Assassin's Creed where you can just grab everything. They start in on textures and texture pops, none of which I've ever seen, and even on PS3 (the PC looks better), haven't noticed. Maybe they're stuck on next-gen mode, when this is still last gen. They sort of half-praise the combat, then critique it somewhat validly. Then go right off the rails with erroneous statements about how its impossible to stun enemies with block counters or any attacks. They close out with some opinions on the story which are valid enough.
Having read that, I can empathize with the guys opinion. They blatantly get several not-obscure facts wrong about the game. They open by it calling it clunky, but then later complement the core mechanic (combat), outside of apparently being terrible at it or not paying any attention in the tutorial by what they got wrong, and then cited as faults. Even with rampant errors, I don't get a "subpar" out of their article overall as written.
Not to be unobjective myself here, the Toronto Sun (of all places), actually (after some brief ranting about how this Dracula is not their Dracula), covers its critical points fairly well, without peppering misinformation throughout. Reading less like someone with a grudge against the rebooting (or Konami itself) scrabbling for justification, and more like someone who legitimately disliked the game.
To borrow some phrasing from a webshow I watched recently, I think part of the problem is there's a growing crop of reviewers who are also dabbling (or throwing themselves wholeheartedly at) also being entertainers with distinct personas. The ranting cynic being an overly popular one. Its creating a growing distortion and negative slant on the whole process. Yahtzee, AVGN, or the Spoony Experiment, and similar affairs are some flagships of the "Comedic sketch of a review" format, and it seems to be seeping into regular reviewers. We're getting more reviews that seem like they're looking to establish their entertainment persona, with less focus on the actual reviewing behind it. This is highlighted when entire articles go into rant mode on one segment of a game, and ditch the entire other 70-90% into a single sentence.
"a game with this quality" yeah... Considering that even as I wanted so much to like the first game, or enjoy it even, I couldn't, which is why I didn't expect anything from the sequel, but having a Dev saying "MY GAEM IS GOOD Y U NO LIKE IT!" is actually pretty funny.
To be fair 7/10 tends to get used as average and underwhelming while 4/10 is usually reserved for games that are actually badly broken so I can understand him being frustrated. Although not everyone dose use it that way which is one of the limitations for metacritic scores. Dose Edge do it differently?
you put a lot of effort on your work and its normal to feel hurt when it seems people dont appreciate it, but dont act like this, dont expect people to say your work is good when its not, how can you expect to ever get better if you cant get a proper feedback on what you did wrong? as highly as you think of your work you are probably the last person to review it, sure you made it, nobody knows it better than you, but at the same time, being something youve invested so much on you are less likely to give it an objective critique
I'm not wrong (you're very good at repeatedly touting your own percieved correctness though), we're just attacking the issue from different perspectives and philosophies.medv4380 said:
The problem lies with the outlets who spent years being overly generous with their 8s, 9s and 10s (which seems to be a slowly disappearing trend now, thankfully).
Yea, you still are wrong. You've changed your position at every turn in an attempt to make yourself sound right. All you've accomplished is proving how much you don't know about averages, and scoring. There is also no different Philosophies unless you call goal post moving a "Philosophy". You had the "Philosophy" that the Middle of a Scale is the "Average". There is no mathematical formal or informal philosophy that justifies doing that. Trying to present it as such also makes people dumber since they may actually believe the falsehood. If you want to present your "view" as a philosophy you're going to have to do much better at forming your opinions. You had the "Philosophy" that what someone says is average is average for them which is factually wrong, and can be proven as such by actually taking their average and showing a single person who says 5 is average when their average is really different. Sure someone might say their average is X and be right because they took their average and are reporting it, but their "Opinion" of what average is doesn't actually make it their average. Now you have the "Philosophy" that we just have different philosophies, but we're actually arguing the Philosophy of Mathematics which happens to be the most formalized philosophy in existence, and as such you can't just take an unsupported philosophical mathematical position and call it your "philosophy" to give it more weight.Steve2911 said:
There has also been no "trend" of giving an over abundance of games 8s, 9s, and 10s. It's trivial to look up and confirm, and if you're going to claim otherwise look up the stats and figures and prove your point. There is a subtle difference in reviewer scores vs user scores at the high end mostly caused the the law of large numbers since the number of users far outweighs the number of critics. This issue with critics having a slightly higher average than user scores is that critics have an issue giving out any score less than 6, and a complete phobia of giving anything a 3 or less. Users don't have that issue mostly due to not having to worry if their review will get them fired. Users give about a 7 and critics, in general, give slightly higher. Now there is a slight difference in the Mode, but not anything of significant. Heck when you look at it system by system that difference flips around for the Wii titles with critics having a slightly lower mode than users. I can give far more exact number if you want.
That is exactly how I'd describe Skyrim, and it seems to have gotten 9/10 from most reviewers. This is with fully patched DLC that still generates consistent (6 out of 7 trials) game-breaking bugs during the main DLC story missions.Karadalis said:
I kinda think that that kind of thing, charging extra money for stuff that doesn't work, should incur some sort of fine or jail time. Maybe I'm a little extreme. But I think we can at least agree that it should result in quite a bit less then 9/10 for review scores.
However, we do know what caused the discrepancy between critic reviews of skyrim and user reviews of skyrim. It's that the Critic only got 360 versions of Skyrim which didn't have nearly as many bugs as the other versions. The Absolutely unacceptable version of Skyrim is the PS3 version, and never should have been released. The bugs also don't immediately show up. They slowly creep in until it becomes unplayable. Since critics rarely play the a game from beginning to end they don't normally see mid to late game bugs. It would be nice if critics would indicate how many hours they actually played a game, but I'd say they play less than 20, and some much much less than that.McMullen said:
From what I've seen with scoring I'd say the longer you play a game the more likely you're to give it a lower score. There is a logical reason for this. Large games like Skyrim are bound to have a few untested game breaking bugs. "Open World" games tends to have this problem in spades. When you look at the metacritic scores for Critics and Users the first day of a games launch all of these reviews were done with the minimum time in the game. After that they tend to fall forcing the average down 1 out of 20 points or 1 of 10 points of what it started out at. People who review a week or two after launch have a much higher chance of encountering a game breaking bug, or just getting made because of a particular level design. Developers will also have removed most if not all of the easy to encounter game breaking bugs in the first few levels because they most likely made those levels first, and via the virtue of being the first part of the game, has been tested the most.
Then you have a games like Sonic that have the exact opposite problem. Critics are normally a full point lower than users on those titles, or they're exactly the same. I'd say the issue is actually that games like Sonic are easy to grasp and understand. If there is a game breaking bug or game mechanic it shows up early and stays the entire game. Another example would be the Disgaea series. This results in the critic who doesn't play for more that 10 hours can now get an accurate view of the games quality. However, this then shows off the selection bias of the users that is a result of the marketing. No gamer ever buys a game that the trailer made them not like the game. They only buy it if the trailer, and marketing, actually made them think it was interesting. So you end up with an over selection of users who know they they will enjoy the game, and an under selection of users who know they wouldn't like the game. For example, if you didn't like JRPG's would you buy a game that the trailer screamed JRPG to you. Your marketing can also backfire in attracting the JRPG crowd. If you get a bunch of players interested in a game that think it's in the JRPG style and it's not then they could out right hate the game much more for not meeting their expectations. A good example of this is actually a movie, The Purge. A lot of people went to it expecting it to be some new take on horror, and instead got a standard home invasion horror flick. Critics don't exactly suffer from that problem, they can, but because they might not have even encounter the marketing by the time they get the game, and are simply playing because their boss wants a review ready by a dead line, and even when they do they can't exactly choose not to review a game their boss told them to.
Here is how I see the scores.
If Critics are Lower than the Users then the game could be a niche title that whether you like it or not should be so obvious. Or the title has some issues early on, forced stealth, that angered the critics, and caused them to view the game differently then someone who continued to play past those parts.
If Critics are only 0.5 to 1 point higher than users then it's normal.
If Users are lower than critics by more than 1 point there are issues that are not readily apparent in the first few hours of play.
Then there are critics who are gaming the system. They've long since figured out that Human beings talk more frequently about negative news then they do about positive news. This is probably why humans keep constructing scales, and then use the first half of the scale to reflect the abysmally bad. This results in bad news becoming click bait when it's the only negative news there.
You can discover who is doing this by using the Law of Large Numbers.
The expected result of a critic is their average should approach the overall average. Unless something is wrong with the critic. An honest difference would be a food critic who is a Super Taster would consistently hate foods the average liked, but would be inline with the average of other Super Tasters even if you think that their opinion is "subjective". In this sense the more "subjective" the more likely the LLN applies since subjective is just a random element, and that's what the LLN is all about. Averaging mitigates the effect of a random subjective element.Wiki LLN said:
For example The Escapist's profile on Metacritic actually shows the measure of this with the point higher or lower score given to critic profiles. The Escapists only gives 0.12 of 10 points higher on average on a sample of over 400 games. That is what the LLN is all about. It shows that the Escapist is in line with the expected results even when they might give a game a 9 that averaged an 8 they'll give another game a 5 that should have got a 6.
However, even though most fall in line, there are going to be some exploiting human negativity. The Edge gives about 0.88 of 10 points lower than the average with a sample over 2,400 games. That is out of line with the LLN and warrants looking into. If they have a number of reviews with simple facts being way out of sync then they would be generating click bait rather than honest reviews.
