I'm torn as well. I can see the need some people have for acceptance and wanting to be where they want to belong. Good! Great! And we the normatively gendered should be supportive.
Having said that, I would worry about those who simply use the trans identity as a cover. What if a guy wanted to spy on women and just said "Oh, I self identify as a woman, therefore its okay." or vice versa. Now of course the overwhelming majority of people wouldn't abuse that system, but some definitely would. And all it takes is one case and boom, management and the media come down hard on the trans community. It'd be an absolute shitstorm and everyone would be pointing fingers, wondering whose to blame for letting a flasher into the women's restroom.
I mean I go to a gym and when I'm in the locker room, I find plenty of unlocked lockers with stuff in them. Open, see things, close, move on to next one. But to an outside observer, it does look like I'm checking out whats in the lockers. Imagine women if you come out of the showers at your local gym and find a guy looking through all the lockers. You gasp. He looks at you, smiles, then returns to his work. Dare I say you'd be concerned?
But I still see it from the other angle too. Trans people want to belong and we should let them. But I don't know how to square the circle of worry, irrational or not.