Desegregating sports would exacerbate the problem.If the problem is cultural it can not and will not be solved by needless segregation.
You don't do anything physically in chess. You can instruct someone to move the pieces for you if you didn't have limbs. Doing well in chess has nothing to do with how you move the pieces. Even in video gaming, there's at least some physicality to it with hand and finger dexterity. I'd consider video games a sport but they'd be as far down the spectrum as you can get to not being a sport but still technically a sport. Chess isn't a sport just like Axis and Allies isn't a sport just like DnD isn't a sport.But it is a sport: and it's a sport for no less a reason than so many people and official bodies recognise it as one.
In a similar vein, as we're on a gaming website, I would suggest you consider very carefully the fact that gaming has "eSports".