I think ancient myths are an important backhistory to the genre, even back in Norse or Greek times, they contain an idea that there was a time of magic and legends, and what humans have steadily brought to the world is mundanity, and the remnants of the old days dwindle and disappear. Even without elves and dwarves, fantasy heavily leans on ideas of a historical golden age from which there has been a decline to the current day, and the modern protagonists are scrabbling around trying to find lost artefacts and knowledge from it.
The creatures of an age of magic in fantasy generally do seem to have treated the humans as a lower class of creature. However, the humans do not replace them by fighting them, it's the the likes of the Elves and Dwarves exhaust themselves fighting various mighty creatures (each other, dragons, demigods, whatever), and once all the contending parties have devastated each other to the point of neat-extinction, all humans have done is fill the depopulated remains of the world.