For me, a large chunk of it is player base.
In my early 20s, living in a college dorm, we played a lot of fighting games. We had at least 20 people that lived within 15 yards who would stop in and play a few rounds. Talk trash, laugh, and unwind from the day.
Now, in my mid 30s, there is a much smaller base. My wife and I will occasionally pop in the last Soul Caliber we bought, and if we have company, sometimes Smash Brothers Brawl, but that's about it. When my kids get a little older, they may start playing some, but right now, there's really just the two of us.
As mentioned above, almost the exact same thing happens with our racing games... oh, and Guitar Hero.