Absolutely it matters. Just with film and TV, background music and sound plays a huge part in your appreciation of a game.
Ther are exceptions, of course. Something like Pacman doesn't really lend itself to an epic, sparing soundtrack (though try playing it with the Benny Hill theme going, it's a hoot). I'm not talking about this sort of game, I'm on about the Final Fantasies, Uncharteds or God of Wars of the world.
Music helps create atmosphere, and for me at least, it's also something of an emotional tie. I can't make it through FFVI's opera aria without sobbing for example, and it's entirely because of the music.
I think in a lot of cases, for film, TV and games, you almost don't notice the music, as it does its' job in a very subtle way. But if you were to see / play it again without any music, it wouldn't be anywhere near as good. I mean, imagine Star Wars without John Williams' soundtrack. You can't, can you?
If memory serves, Buffy did this once, when her mother died. No ambient music for the entire episode, just what people said and any noise their environment made, like closing a door. It was designed to be weird and chilling and it sure as hell worked.