Part of it may be generational/geographic. I feel as though in the 80s and 90s (when I was in K-12 education), there was a definite jocks vs. nerds divide with a lot of bullying and harassment. I got some of it, but far less than many as I was always a pretty big guy. It's a little humiliating for a jock to run up onto a table of 8th graders playing D&D, kick some books over, but then be hauled down and punched in the face by a DM who was a good 6 inches taller. I feel like the mix of anti-bullying efforts and nerd culture becoming mainstream has lessened that (though not eliminated it) today. Finding a jock who doesn't also play video games and go to comic book movies is a lot harder today; you can probably find some who enjoy manga or anime without much effort.
But I think the bigger part is the general rejection of something popular if you don't agree with it. Look at the anger against the pop star of the moment, all things Frozen, games journalism that includes critical analysis of the content, or even sports that the person in question doesn't consider a True Sport(tm). Soccer took a long time to overcome that.
SO I think it's a mix of remembered (or present) school age bullying by "jocks", a general disdain for something that is found to be boring in the face of those for whom it is a major event, and I think, a bit of elitism as sports fans tend to be seen/portrayed in problematic ways. Just look at marketing during your average night football. From the amount of beer, viagra, and truck commercials, even the marketers seem to think that football fans are all impotent alcoholics who want to pretend to be cowboys or construction workers.
Personally, I find watching other people play a game to be boring as all get out, whether I'm watching the Superbowl or a global League of Legends tournament.