I would be quite upset if there was no such thing as easy mode any more. Certain games that I'm not used to require me to start on easy mode, to get used to controls, enemies, etc.
To give an example, I recently played my first Metal Gear game ever, Metal Gear Solid 2 (I watched a let's play of the first one, so I wasn't entirely lost plot-wise). I started on Normal because I was trying to act tough. Unfortunately, the Metal Gear Solid games' controls are so different from any other games that I had to stop that playthrough and switch to easy because I was getting my ass handed to me due to my lack of knowledge of just about everything.
So no, I don't think the easy difficulty should be removed. Because you can never know when somebody new will start up the game and get destroyed due to lack of knowledge.
That being said, I have no problem with games that lack difficulty levels at all. Usually those games are designed in such a way as to be easy to pick up but difficult to master. That's fine.