A small piece of candy that usually comes in a box. Very crunchy. Oh, wait. Wrong type.
My definition of a nerd is someone who fully enjoys talking about a certain subject. Like they could talk for hours on it and not grow tired. There are math nerds, science nerds, Star Wars nerds, video game nerds, My Little Pony nerds, all sorts of nerds. These are people who are a step 'above' (because I can't think of a better word) your average fan. There are people out there who enjoy Star Wars, and then there are the people who dress up or who can name the founding members of Luke's Jedi Council without thought. You know you're talking to a nerd when you see the look on their face. The sheer joy that someone has asked them about their subject. It goes beyond bragging, beyond explaining, beyond talking. It's a completely different look. I am a Star Wars/video game nerd myself.
As for the term 'gamer,' I've always applied it to people who are immersed in the gaming world. People who get the joke of your T-shirt, or who understand what you're talking about when you make an off-hand reference to a video game. I don't consider someone who plays Call of Duty or Battlefield for hours on end to be gamers. I consider those people as people who play games. Or a game. Sure, they may be really good at their game of choice and able to stomp me in it, but I don't see them as gamers. I can't ask them, "What did you think of this game?" or "Have you played this?" because they'll respond with, "What's that?" most of the time. Not a gamer.
Does it really matter? Well, sort of. As you said, you feel left out because you never got that title assigned to you. People will often do this to keep outsiders out of their group, or at least be able to identify other people they believe they can talk to. It's not that hard to fit in though. All you have to do is find people with the same interests as you. You may also have to come up with a difference between a 'geek' and a 'nerd' because that can get confusing...for some reason.