Very, very rare is the person why actually has a problem with casual games. The miscommunication here is that Nintendo, lately, has thrown the idea of innovation completely out the window, in favor of bland, safe, shovelware games from third parties, and just non-innovative tweaking of safe franchises for the in house stuff. Casual games are awesome, I think we can all agree. But poor quality casual games just lower the bar for everyone.
Casual games have become synonymous with the Wii, and the Wii's casual market has become the standard. Unfortunately, useing the Wii as a standard is really, really bad. After having there most productive year yet in software, I almost run out of fingers counting good Wii games. Amongst the good Wii games, about half of them are adult, non casual titles. Then there's ports and re-iterations of established franchises. Wii casual games are doing little to nothing in advancing game design as a medium. There just cashing in. This will catch up with them, in time.
Just to establish something, I have a Wii. It was not a waste of money. I am playing Muramasa now, and have both the new Silent Hill and Dead Space coming my way. It is a lot of fun. But while PS3 and 360 pump out new ideas all the time, I can get everything good on the Wii with next to no effort. This gives casual games a bad name, because the ones so in the spotlight are frequently trash. Casual games though are great. Popcap are masters in design, a lot of the best Indie games are casual, and with things like PSN and the XBL Arcade, casual games add a whole, exciting new dimension to modern day gaming. Its just that the casual game spotlight is squarely placed on the sub-par Wii games.