How do we judge old games? With Nostalgia Goggles on, of course!
but, yeah. Graphics do matter. Early 3D was fugly, even compared to some 8 bit games and there's no denying it. It was an important step to take, but it doesn't change the facts.
Difficulty... is a trickier issue, because the culture of video games has changed.
Difficulty wasn't just a relic of arcade machines gobbling up quarters, it served a purpose by giving the game longevity. If you blasted through all 10 levels of a game in an hour on your very first run through, you might feel cheated. If you have to slog through it over days and weeks of practice, then you feel you've accomplished something.
Also, in the olden days it was harder to tell a good story, so you had to make the gameplay challenging to keep people interested. Nobody played Super Mario Bros. for the story, and if you breezed through it on your first run through you'd say "meh, no big thing", not "OMG this is an iconic game that changed the face of gaming for all time!".
Old games that had a decent story tended to have more forgiving gameplay, saves and level codes, so that you could keep the story going.
Now, games are nerfed so that everyone can make it all the way to the end and see the whole story... because the devs think the games actually have a story worth telling (they're not always right.) And because we expect to get a lot of content for our gaming dollar.
TL;DR: Graphics matter in context, Games were difficult for a reason: extending game life.