If you think it is rude, it is rude. If you do not think it is rude, it is not rude. This is true of all manners.
European culture is different from Asian culture which is different from African culture et cetera. Using English table manners in China could be considered rude and vice versa.
If someone does not remove their hat somewhere you feel they should, unless they are aware of your expectations and deliberately flout them, that is not them being disrespectful, that is you interpreting that inaction as disrespect.
I have a general method for conveying respect: I speak to them as though they were my equal; I consider their opinions as deeply as my own; I adhere to their customs while in their home and cater to their comfort while I am their host. Thus far, this method has proven universal and effective, and I have not had to worry much about hats.
If you look to take offense, you will find it.