I think it was the general feeling of it being unnecessary.
Avengers 1 was cool because we finally got to see all of these characters on-screen at the same time, after 5 films worth of buildup.
Avengers Infinity War and Endgame were cool, because we finally got to see the payoff of the Infinity Stones, and Thanos, after about a billion films worth of buildup.
Age of Ultron was just a sequel. It was just the Avengers coming together again. There was no real buildup, there was no real payoff, there wasn't really much of a twist to the normal MCU formula, and there weren't any cool gimmicks. It just ultimately felt like "Avengers made a lot of money, lets make another one". The same can also be said about a lot of the other mediocre MCU sequels.
Also, there weren't really any new hero interactions. Sure, we got introduced to Scarlet Witch and Vision, and War Machine and Falcon basically made cameo appearances, but out of the proceeding movies, only the Guardians of the Galaxy were new, and they weren't even in the film. It was the same Avengers crew, and friends, basically.
Was Age of Ultron a better movie than Avengers 1? Maybe. Probably. But regardless, Avengers 1, and IW and Endgame just felt more special.