Harems, the way they're pictured in things like anime, are a fantasy concept. Life is more of a reality deal.
In my experience, any attempt to blend the fantastic with the realistic ends in misery.
As to whether I'd want one? Well, I see the appeal of the fantasy. It's reaffirming to have multiple people who accept you and find you desirable, after all, and a multitude of lovers stimulates the caveman portions of the psyche that associate sex with masculinity and power.
Still, there are a lot of real-world ramifications to the harem setup that feel like they would eventually end in tragedy. For one thing, aside from a little "shoujo-s" (uggh, I hate that concept) fanservice play between the members of a harem on occasion, they all invariably have eyes only for the main character to the exclusion of the rest of humanity. The main character, however, must remain non-commital for the harem format to work, giving teasing attentions to each girl without special thought to one above the others.
This is a non-equal level of devotion in the relationship, which means at some point or another hearts are gonna shatter.
On a less psychological level, when I first moved in with my girlfriend, the strain of living with someone new for the first time was already a bit of stress on the relationship until we got used to each other, and there was just one of her! the average harem has five members, daily life in that situation would be rough.