To put it simply? It's a character that ends up embodying bad writing, and I mean that both figuratively and literally. Figuratively in the sense that at their most obvious Mary Sues/Marty Stus tend to be less developed characters than a wish list of awesome abilities or features molded into human form, literally in the sense that the story tends to be about how awesome these characters are, and that's treated as the damn draw. And before anyone asks, yes, the Isekai/Portal Fantasy/Reincarnation 'genre' is packed to the brim with these. It's not ubiquitous, but the 'genre' does have a disproportionately high concentration of such protagonists.
My basic benchmark is "does it read like 'My first OC'?" You know, immediately hits it off with the main cast, possessed of rare abilities that make them more special, instantly takes to any role they have to fill (and often shifting between several), hints at a special heritage to evoke the 'secret prince in exile' convention, supposedly has some inner darkness or affinity for evil magic that isn't reflected in them (outside of possible broodiness) that sympathetic characters will say doesn't matter and unsympathetic ones will hold against them, blah blah blah, etc etc. Mind you, I'm not talking about characters who have any one of these. I'm talking about characters where these are piled on like logs on a bonfire.
Basically the benchmark is that ruggedly handsome Grey Jedi who is wholly good (if a bit grumpy) but whose enlightened views allow him to wield Force Lightning Electric Judgment without any fear of falling to the Dark Side, and who practically ends up being a Jack of All Trades with abilities approaching if not surpassing those of specialists. Oh, and also he probably either has secret familial ties to the legacy cast and/or is going to end up romantically involved with one of them. It's the bad "I'm worried my character is not special enough" OC. You know the mold.
But here's the catch, that mold is just the red flag. The Mary Sue archetype is more than just the character itself, it's how the character is used in the story. The seemingly perfect character that everyone fawns over is terrible as a protagonist that we're supposed to empathize with, but works spectacularly well as a rival or antagonist, because that changes the gears on the story from "look how awesome my Original Character (do not steal!) is!" to something that the protagonist(s) have to overcome.
For the sake of example, the second book of Mistborn introduces a new morally ambiguous character that manages to routinely go toe to toe with Vin, is obviously intrigued by her, and is implied to have taken a romantic interest in her. Moreover, Vin herself repeatedly wonders if he's a better match for her than her love interest. Truth be told, if we look at his character in a vacuum, he hits a lot of the Mary Sue flags. However, he doesn't really come off as one because the story is not about how awesome he is. He's an antagonist who practically embodies Vin's doubts and insecurities about her place in the world. He's the temptation to give up and run from the people she cares about, because she's afraid that - one way or another - she doesn't actually belong with them and that by rights they should reject her. He does not exist as an awesome [side] character we're supposed to root for, he's an obstacle that Vin must overcome as part of her character arc, and that saves him from being a Mary Sue.
Contrast this with Wesley Crusher who got hit by the Mary Sue truck so hard in the first two seasons of Star Trek: TNG that it's a wonder that he didn't become an "Isekai" protagonist. So much of the writing around that character boiled down to him being some praiseworthy genius (And I mean that the character is literally called the math and physics equivalent of Mozart) or being unfairly not praised for his genius, to say nothing of the of "Wesley saves the day" stories, which could border on him being almost messianic. Heck, his final bow in the series was leaving with the Traveler (kinda a Q-lite, not omnipotent but freely capable of manipulating space and time with their minds) to develop his own Q-lite abilities (which he has simply because he's a super special human). And the start and end point of that basically boils down to "Wesley's supposed to be awesome". And he's hated for it (including by Will Wheaton himself).
(Edit: Typos)
