It depends on the game.
Some games are best with 2-4 people. Split screen games, co-op, some racing games, etc.
Frag fests are best with 8-16 players. Fewer and the action will be too slow and less entertaining, anymore and it'll be too crowded and you'll die moments after spawning.
Some games are best with 32-64 to even 100 players in some cases. Usually big, wide open battlefields with plenty of travel time and strategic opportunity (Battlefield series, MAG.)
And then there are MMOs, which are best with at least over a thousand players in a persistent world so that you have plenty of people to trade/buy equipment with, form parties, guilds and roleplaying scenarios with, etc.
It really does depend on the game, there is no "too few" and there is no "too much". I don't get what you mean by the "new 4". A decade later, there will still be some games that are best played with 2-4 people.