Let's look at the different ways that I've heard people ask that, because sometimes, it can be a very loaded question.
1. The Legit (And Sometimes Flattering): I think almost got this question a while back because I was talking about how I liked Glee in a GLBTA meeting. Then I mentioned my ex girlfriend. A guy that I was getting along with looked disappointed, but then we went back to joking around about the fact that we just saw the guy that did Zuko's voice making out with another dude in a movie.
In this case, the question would be totally justifiable, because it's legitimate.
2. The Legit (But Insulting): I got this quite a bit when I was younger. I used to be... let's say "naive", and I thought I wouldn't have sex until I was married. When I mentioned that, one of the jocks looked at me and said "are you gay?" to which my response, at least nowadays, would have been "are you implying gay guys don't have sex?" to which I would have probably gotten a hell of a confused look.
I also feel like I got more than that guy ever did, because god damn was he ugly >=) anyways...
In this case, it's obviously offensive, mostly just to the person being asked it, but also to homosexuals, because it implies that there is something wrong with both parties. Maybe it isn't purposeful bigotry, as the guy was young at the time (we were 15), so it could just be general ignorance. But it doesn't make it any less wrong.
3. The Joke ("Hah Hah"): I have an openly gay friend. When he was talking about musicals and theater, I asked "what are you, gay?"
He said "duh, but don't bother asking me out, I have standards".
We laughed our asses off. Therefore, obviously not offensive. But that's only because I'm good friends with him.
4. The Joke (... Really?): Before previous said friend was openly gay, he was in the closet throughout his high school career. Once, someone made a joke and then ended it with "so I asked him, 'what are you, gay?'" and it came off as EXTREMELY homophobic. This is, in my opinion, the worst kind, because the guy was actually homophobic. He 100% meant "gay" as an insult to homosexuals.
So, offensive. Duh.
5. The Abstract (Grapes): A Bronie once asked me if I was gay. I said "Only in Wonderland, but I've never been, so I wouldn't know." Then, we flew off to Detroit to watch the cast of Community face off against the New England Patriots in a game of cribbage.
So, that one is bananas.
But those are just my personal experiences.