It frustrates me to see people trivializing this issue by saying things like "Why don't you play on peaceful!". That's missing the point, because this isn't a difficulty issue, it's an aesthetics one. The mechanic doesn't effectively work as intended to increase difficulty.
The idea of an Enderman breaching the wall of your sanctuary to let hostile mobs in is nice and all, but how often does that actually happen? That implies they have the intelligence to actually make a hole that something can get through. They don't. The reality is that they can only achieve this through dumb luck. They don't actively make an accessible hole, they just break random shit. Typically, this is annoying and unsightly, but rarely dangerous.
Does having to make daily repairs to patch up such ugly but otherwise inconsequential vandalism make the game harder? No, just more frustrating. And no, Creepers can not be compared. When a Creeper explodes, it does so in your presence, because you failed to prevent that (therein lies the challenge), and you can repair the damage because you know what was damaged, you saw it take place. Endermen are far more annoying because they vandalize things even when you're not around to know about it. They could only be compared to Creepers if Creepers exploded randomly when they're nowhere near you, leaving the landscape looking like it'd been hit by an artillery barrage by morning.
And it's not just my stuff I'm concerned about. Endermen's destructiveness extends to the natural world. They spoil the unspoilt wilderness. Plains look like a pack of prairie dogs got to them (holes everywhere), and forests look like some idiot came through who couldn't be bothered to cut down trees properly.