It doesn't make you a terrible person but it doesn't make you any kind of good person either. Neutrality is neutrality, and so long as you and anyone else sits on the sidelines, insisting that games/movies/whatever are just entertainment, you aren't doing anything praiseworthy. Don't be surprised if those who DO care call you a jackass either.
These problems keep resurfacing, and the only way to stop them is to consistently call them out. Staying on the sidelines doesn't help that cause, and in fact only serves to help keep the shitty game development industry shitty.
The fact that you are a man affords you the ability to not care, but for women/PoC/non-hetero/non-cis people, it's impossible NOT to notice that you aren't represented, or that games actually actively support your objectification. Even if we want to ignore it, we often can't do so without effectively ignoring who we are, suffocating our identities just so we can play that fun game about fast cars and explosions.
So yeah you can not "care" all you want, but that kind of just makes you an asshole. You're the kid in elementary who's mom packs him a super awesome lunch every day and just can't fathom why all those other kids are constantly complaining about the terrible cafeteria food, and wants them to shut up about it.