Bluntly, if you stop to think about it for more than five seconds, that argument is incredibly inconsistent.If you'd like to make the argument that if a person has their penis removed it makes them no longer a man, that's an argument made by lots of people with various intended meanings. "Manhood" is a term for male genitals for a reason.
But that's not really the point of contention, is it? Extraordinarily few people care about the things people claim not to be, the argument is largely about being something else. A trans-woman not being a man isn't really the argument, it's being a woman that's the sticking point.
For virtually all practical purposes, sex relates to physical characteristics, specifically the presence of [external] primary and secondary sexual characteristics. The sex you're assigned at birth isn't based on some comprehensive dna scan, chromosome analysis, or anything so complex. It's literally decided with little more than a glance between your legs. If you have a penis, you're a boy. If you have a vagina, you're a girl. Simple as that. Hell, in the case of the intersex, SOP is basically summed up as "close enough" based on what their...equipment most resembles. "I think that mostly resembles a vagina...so congratulations, it's a girl! Don't worry, we can make that look more like a normal vagina."
But when we talk about trans individuals, suddenly the rules change. Sex is suddenly has nothing to do with their morphological characteristics, but instead is entirely about whether they have the right chromosomes, or whether they have functional ovaries or testes (which comes packed with a backhand to those who are infertile or have had a vasectomy or hysterectomy). Let's not kid ourselves here, we change the rules for little purpose other than to...well, 'other' them and claim that they are not and indeed can never be that sex as a matter of definition...a definition we don't otherwise apply.
In effect, we're saying that the standard rules don't count for them. Everyone is assigned a sex in the delivery room purely because of their morphological characteristics and the expectation that those reflect their future morphological characteristics. But after they transition? Those same morphological characteristics are suddenly irrelevant, and it's instead all about the chromosomes that nobody ever bothers to check unless testing for congenital issues. Don't believe me? Do you know when Klinefelter Syndrome is diagnosed? You know, when they realize that the kid had an extra X chromosome and was XXY instead of XY? Puberty, after they realize that the testicles have failed to grow. Similarly XX male syndrome (Morphologically male from birth despite having XX chromosomes) is typically recognized in late adolescence with the smoking gun that leads to the chromosomal test being - again - small testes or infertility. Let me reemphasize that: the first time they learn that they don't have the typical XY chromosomes is during puberty when the doctors suspect that the root cause of visible issues is a chromosomal abnormality. And that rather neatly illustrates how chromosomal testing is the exception, not the rule.
You weren't assigned male because the doctors performed a chromosomal analysis on you in neonatal (in fact, the balance of probability is that they didn't perform such an analysis at all). You were assigned male because the doctors saw that you had a pee-pee. Neither would it have mattered if you had turned out to be infertile or had otherwise lost function in your testes. You would still be considered male, because lacking reproductive ability does not disqualify you under the normally used definition.
There's no scanner at the entrance to the men's room that checks your sperm count or verifies your Y chromosome and testosterone levels. You don't have to submit a government issued ID to prove that you're male, society operates off a "you know one when you see one" system. If you're joining the men's swim team, they don't force you to prove your maleness with genetic testing. Hell, maybe it was different for you, but I certainly didn't have to provide a doctor's note or drop trou in the DMV to get "Sex: M" on my driver's license, nor did they insist that I submit to a genetic test to prove my sex when I checked that box on the form. So why do we insist that genetic profiling and reproductive capacity are the only relevant factors when talking about trans individuals?
It's equivocation, pure and simple. We use one definition most of the time, then swap to another to create a pretext to exclude a group that we don't think should qualify but couldn't exclude when utilizing the usual definition.