Endermen are weak, if you have a bow they don't even move when you shoot them. Their block taking ability made them at least interesting, don't like it, just add a moat.
Make Endermen teleport the player to them instead of teleporting to the player, THAT would be scary as hell.
Make Endermen teleport the player to them instead of teleporting to the player, THAT would be scary as hell.