Not necessarily, no.
Biological sex is usually categorised according to several factors: gonads, genitals, hormones, and/or chromosomes. None of which are 100%.
To say that "trans men are women by sex", you would have to either 1) believe that chromosomes are the sole and overriding factor, or 2) be going solely by sex at birth, regardless of whether those above factors have changed.
It should be obvious that neither of those options actually hold much water when you look a little deeper. Chromosomes are functionally the least useful indicator of the four, for instance; they cannot be changed, but their sole function is in forming those other physical characteristics... which can be changed. Plus, throughout human history we have categorised the biological sexes without ever knowing chromosomes existed. And still, today, doctors and veterinarians determine someone's physical sex without looking at chromosomes.
And there's just no rational reason to consider "sex at birth" to be the same thing as someone's present physical sex. Scientists don't. Doctors don't. You might as well consider someone's height at birth to be their "physical height" regardless of what changes when they're an adult.
The anti-trans brigade like to believe they have science on their side, but they're actually a bunch of scientifically ignorant tools.