If the work in question is a multiplayer game where you have to actually interact with the fanbase in order to interact with the work, then it would make sense to avoid it if the fanbase is horrible. But in any other instance, letting the fanbase affect your perception of the work would be utterly irrational. Understandable in some cases due to the association, but still irrational.
If the work is something you'd enjoy otherwise and you don't have to interact with the fanbase to experience it, then you'd be doing yourself a disservice by not making an attempt to get past that mental association. If you're afraid of being associated with the bad stereotypes around that work in the eyes of other people, then you can either simply try to keep your enjoyment of the work quiet, or every time you mention it, make a point of specifying that you're aware of the unpleasant part of the fanbase and that you're not a part of it.
If the work is something you'd enjoy otherwise and you don't have to interact with the fanbase to experience it, then you'd be doing yourself a disservice by not making an attempt to get past that mental association. If you're afraid of being associated with the bad stereotypes around that work in the eyes of other people, then you can either simply try to keep your enjoyment of the work quiet, or every time you mention it, make a point of specifying that you're aware of the unpleasant part of the fanbase and that you're not a part of it.