Queen Michael said:
Am I a bad person for disliking him?
See, I think we can agree that it's absurd to say you can't ever dislike "special needs"-people. The idea is that you shouldn't dislike them simply for being "special," and that's a perfectly fine sentiment with which I agree. Thing is, every part of their personality is created by their being "special," or to put it differently, they would be entirely different people if they weren't special. So is it ever okay to dislike a person who's "special"?
Of course you can dislike people, no matter what their intellectual capacity is.
I see from your profile that you are Swedish - is that correct?
If you work in Sweden, then this trash collecting person is probably employed by a company that provides meaningful work for people with intellectual disabilities. The biggest is Samhall, but there are others. This means he has a supervisor, and part of that supervisor's job is to help deal with the issues that occur. The fact that he acts inappropriately is such an issue. If you talk to him or to his supervisor about this, I think they would appreciate it. They need to know these things, particularly about the pushing. You seem like a nice person, but others he annoys may not be so nice. This is an "arbbetsmiljöfråga" for you, but for him it's a safety issue.