I didn't grow up with the books, and had no interest in the Harry Potter world, until I saw HP and the Prisoner of Azkaban on television. I stopped and watched it, and immediately went out and bought all the movies that had been made at that time. Since then I've seen all the movies in the theater, and read all the books.
I can't quote lines, and sometimes I forget what happened in what book to what character, but I appreciate it for what it is- An original, creative, and layered world with interesting characters and a good story.
I would consider myself a fan, and honestly I am disappointed that the movies are over, because it gave me something to look forward to seeing the theaters.
When I saw the most recent movie on Saturday, there was a group of people dressed like various characters from the movie. While I was unimpressed with their level of devotion, I can say that they were quiet during the movie, and they spoke in normal tones to each other when entering and exiting the theater, and with a movie with that kind of fan base, that's pretty damn good.
So, no, Harry Potter fans aren't annoying, or at least none of the fans I've seen.