While I can understand the angle you're going for with your comparison, I still think that Fedoras and Miniskirts are apples and oranges. The miniskirt is a common, even functional everyday piece of apparel and anyone with a normally functioning brain wouldn't feel any prejudice towards a girl in a miniskirt.
A Fedora is different for two reasons.
1. It's not a piece of apparel that covers anywhere that needs (in a way that would prevent you from being arrested) to be covered, thus its importance as a piece of clothing diminished already.
2. It was a hat popular in the early 1900s among men that wore suits tailor made to fit their body and fedora and you it looked good on them but not good on some sweaty, fat nerd with a trench coat and knee-high boots.
Perhaps you could compare it to something a bit more in tune with a hat but that girls tend to wear? I don't pay that much attention to fashion but maybe it would be more akin to leg warmers, if they were worn by and used as some kind of symbol by pseudo-intellectual misandrists or something?
[sub]inb4 someone gets offended that I used leg-warmers as an example even though I don't mean it literally.[/sub]
Hell, I don't even know why you targeted this thread at feminists in particular. I've seen people throw the Fedora insult at your typical, pseudo-intellectual "white knight" or "SJW" and they tend to find it just as insulting. I don't think there's really anything about the hat itself that draws peoples' ire. It's just that when you use the fedora insult, people get insulted.
Me, personally. My hat of choice is the beanie. I fucking love beanies. It's too bad I live in the middle of tropical Queensland, Australia and only get the chance to wear them when it gets really cold here or I go to the snow.
I'm not particularly attractive to begin with but I think the beanie suits my face well and it doesn't have any retarded stereotypes that go with it. Also, it looks just fine with casual clothes so I can wear it freely as long as the weather permits. It would look silly to wear it in the middle of the summer, so I don't do that. It's a conscious effort I make to wear a particular type of head wear when it's appropriate. A kind of effort that your typical steaming neckbeard doesn't make when he's wearing his fedora with clothes that don't match.