I think everyone is dancing around the point. Dictionary definitions of "game" are irrelevant; I don't think any of us play games just because they're in the same category as sports, chess and twister. We draw some value from gaming, a value that doesn't stem just from the rules, the conflict, or the challenge.
Doubting Dear Esther's status as a game isn't a semantic quarrel, just like homophobes aren't really worried about human procreation when they gay-bash. It's merely a case of people being uncomfortable with something that changes the status quo, so they argue semantics when the product obviously still belongs to the same hobby and is played for the same reasons (exploration, emotional investment, catharsis) as all other videogames.
It's also a matter of degrading other people's experiences - if they didn't play a "game", then they don't get to bring up the non-game in question in discussions and "it doesn't count" when arguing about the nature of the medium. It's also vehemently exclusive and imposes one group's definition of the medium ("a game is something where you shoot, punch, jump or talk at things, and watching/moving does not equal interacting") over everyone else's, presumably also telling them that they didn't actually enjoy the thing they enjoyed.