What's nerdy? Wearing a Football shirt, sitting on a sofa, and pretending you know more about Football than the people who do it professionally.
Tabletop wargaming is, like Roleplaying, still labelled with this bizarre stereotype of all its fanbase consisting of pale-skinned, skinny people / fat spotty people who've never seen a girl naked.
None of the people I game with are pale-skinned. Some are verging on fat, but being 40-odd does that to you. All of them, save those who are too young to have done so, have slept with at least one other person (I don't say "woman", because some of the gamers are straight-females). Most have a far healthier sex-life than anyone who follows a "cool" hobby, like getting pissed and yelling at the TV during a sporting event.
Frankly, ANYTHING is "nerdy" when taken to an extreme. It really makes me cringe when someone tries to make fun of me because I can tell you everything you never wanted to know about Doom, yet think it's perfectly normal to have memorised the results of every Manchester United game since the founding of the club. I would argue that video-gaming, however, is far more nerdy; tabletop wargames and RPGs are by their nature a social past-time, whereas videogames (particularly now, not so much in the past) are about sitting alone in your bedroom. Unless you own a Wii, you'll be playing alone - people on the internet don't count.