I can't really tackle "the geek community" because that is a far too nebulous term that means different things to different people. I will however take a stab at answering specifically in relation to gamers.
Myself, I suspect a lack of overlap in interest is primarily responsible, different things appeal to different people with videogames being by and large a storytelling medium (having more in common with movies, comics and literature which is why these things are more readily accepted as talking points on a gaming website). This has little overlap with sports and while some people are certainly bound to find both appealing there is no natural synergy.
In addition sports tend to be very sport-specific. Where, as a gamer you might have a preference for different types of games most people play games from a few genres and franchises and one gamer can usually find common ground conversationally with another gamer, it isn't necessarily so for sports fans. Sports fans seem (as far as I've observed) to usually find "their sport" or occasionally a couple of sports and stick to it. One sports fan who loves hockey has no common ground with another sports fan if they like football and baseball.
Finally I think culture has a lot to do with it. Sport, particularly Football here in the UK, tends to hold itself up as some universally positive, aspirational endeavour. You always hear Footballers talking about how "sport builds communities, sport inspires people to improve their health, sport brings people together". And occasionally it is, in 2012 - 2014 we saw a significant increase in sports uptake in the UK following the Olympics. However, that isn't universal and what sport often inspires is division, violence and vandalism. Hell in 2012 there was even a report released linking the losses of national football teams to spikes in domestic violence incidents.
For me, I have never really had any interest in sport, watching or playing.
In regard to watching, while I accept there is a level of skill on display it simply isn't enough to hold my attention for the length of a sports match. I admire it, but in the same way I might stop to watch a street performer ride a unicycle, its impressive but not something I want to watch for 90 minutes. My ex was very into sport and avidly watched the last Olympics and I tried to be interested and watch with her but I just found it overwhelmingly boring, I even fell asleep twice.
In regard to playing, it just doesn't appeal to me. I walk pretty much everywhere (so far as is practicable) so get enough exercise and otherwise just don't see the point. I don't enjoy it and have better things to do with my time so... yeah.