I dislike the idea of polyamory. However, I do not believe that it is intrinsically and universally wrong. Infidelity, on the other hand, is, and what's more it seems to me that society at large agrees. For most, I think, the typical reaction to polyamory is that it is unnatural, odd, or abnormal, whereas the typical reaction to being unfaithful is instant condemnation most of the time. There are obviously exceptions either way, but the general responses seem consistent to me.
People who advocate tolerating polyamory are understandable, I think. Marriage is an important part of life, and it isn't surprising that people react to it in different ways. Those who apologize for infidelity, however, are wrong, just like infidelity is wrong. There is no reasonable, conceivable, imaginable scenario in which cheating on someone is "right" or in the least bit justified. People who do it because they're unhappy, or "in love" with someone else, or for any other litany of other reasons might have understandable motives, but never excusable ones. Adultery isn't just dishonest for breaking a promise, or a sign of weakness for being incapable of controlling yourself, but it is utterly, unnecessarily cruel. There is no situation in which someone "has" to cheat: doing so is just a selfish indulgence, putting personal pleasure above obligation, and hurting other people for no better reason than because you can. It is evil, and those who argue otherwise are imbeciles.