I think the general consensus is, that prostitution often goes hand-in-hand with crime. Nothing inherently wrong with prostitution itself, but it's a slippery slope to crime, and attracts criminals.
Following that train of thought...
A bit like drugs. As long as an addict has a steady supply of drugs, he's okay. If a guy is living in a mansion, on top of a mountain, and has an unlimited amount of LSD that he only uses himself, he's not a problem.
But when he runs out of LSD, he's going to need more, and he ain't got the time to start filling job applications.