I have a fedora, not a trilby. My usual out-of-the-house style is business casual (long sleeved button up shirt and slacks), so my fedora tends to compliment the look. I accompany this with my leather trench coat, weather permitting. If people react badly to these things, they aren't willing to do it to my face. Further, if they're willing to make a severe judgement on me based on my clothes, they aren't worth having around. Period.
That being said, the "Just A Hat" option is the only reasonable one to pick on the poll. A fedora means nothing in and of itself. It's an article of clothing. My personal opinion is that those sorts of hats look best with conservative clothing or suits rather than t-shirt and jeans, but I'm not the fashion police. If your clothing isn't doing me personal harm, I'm not going to sweat it.