Time is the progress of states; as long as there is change, there will be time, inherent in the ordering of those changes. Your argument is like saying that distance isn't real because a meter could redefined as being 120cm
Here's a fun one though: consider the direction of time. We imagine that time runs from past to future, because we have memories of the past. However, there is no reason why entropy, rather than seeking to average out, should not instead attempt to gather together. This would cause hot things to get hotter and cold things to get colder, and all our physics to run in reverse, including the passage of time.
Of course, the rebuttal is that one has memories of the past and not of the future, but if time ran the opposite way then your memories would be actively dissolved as your body sought to focus all it's energy into glucose molecules.
There's no proof that time runs either way, but it saves a lot of effort just to assume entropy tends to averages.