If the argument is, "Such and such is not MUSIC" then I absolutely agree that they're full of it. Its basically like saying Jackson Pollack isn't painting, which is ridiculous.
If the argument is, "Such and such is not ART" then that's a lot harder to answer. Art means different things to different people, and what qualifies as art is different in people's eyes.
I think, and this is just my opinion, that art is something with a story or a message involved, and that message or story should be, at least in some way, relatively clear. Now, it can be deep, complex, or even convoluted, with multiple layers of meaning that not everyone will understand or appreciate, and that's ok, it's still art. But I think when you start to get into the realm of ridiculous, where the majority of people, even educated people, have to ask the meaning of a piece, then it's not art. Its like when you tell a joke and no one gets it, and you have to explain it, its not funny anymore. It might be clever, and well-thought-out, but if it fails to deliver its message/story in a relatively clear/efficient way, then the whole point is lost.
Again, just my two cents.