It's an interesting question. I had a discussion about this a while back, and I had to come to the conclusion that visual novels with choices are games. If a game where you can become the all-mighty quick load warrior counts as playing, then so does picking a choice in a visual novel - both can be simply seen as going through every option, and quick loading back to the choice, until one finds the correct solution to get to the next divergence. The QL warrior quick saves after successfully dodging, the visual novel-er gets to the next choice, and this is repeated until the QL warrior wins the fight and the visual novel-er gets to the ending. Comparisons can be made to menu-based gameplay as well.
That being said, although visual novels with choices I consider games, I do not consider that you play them, merely that you interact with them. If you split an animation into its frames, a person views the slides, not watches them in the sense that someone would watch the animation in it's intended manner - in the same way, a QL warrior is taking gameplay, but then splitting it into very small parts that it becomes hard to be considered playing any more.
That being said, even the reading is very interactive. You click through every line, and it immerses you beyond using auto read. Consider the knock out animations of Deus Ex: Human Revolution and the knock out animations from the Dishonored E3 presentation, where the player button mashes momentarily to knock the target out. The latter is interacting with the game, even though all that interaction is doing is making the game go through its intended course - same thing with clicking through each VN line. But then it's the same thing as reading an Ebook, thus showing that nothing can be categorised simply.
TL;DR - you are constantly interacting with a visual novel, but interaction is not the same thing as playing, thus I would not consider visual novels things that you play, despite being games.
In case you hadn't guessed, I myself read visual novels. Ones I've read are Fate/Stay Night, half of Steins;Gate, Don't Take It Personally Babe It Just Ain't Your Story, Muv Luv Extra, currently reading the newly released Cinders.