Thing is, you don't get to decide how another person is supposed to interpret the words you use. If you say 'fa*' in front of a gay person and they feel upset about it, "I didn't mean 'gay'" is not a very good excuse.
I absolutely appreciate that you don't use it in front of strangers, but...