Well, it really depends on the kind of game you are making, doesn't it? Let's look at a few examples:
Morrowind: The right size, because it's set in a place known for having large areas of very sparsely populated wasteland. There is a rich contrast between the different outdoor areas in the game, avoiding the issues that Oblivion and Fallout has in this respect. Fast travel could have been more extensive, but I'm sure there are mods for that. And hey, you can levitate. With enough skooma and powergaming, you can almost make your own fast travel system.
Oblivion: Too big. This is mostly because of the scenery and the cut-and-paste dungeons. On the outskirts of Cyrodiil where it borders the other provinces, you typically get a change of scenery that matches said province, but more often than not it's grass, trees and more grass. Fast travel makes it easy to get around, but you'll still be greeted by that horde of generic grass and trees when you arrive.
Fallout 3: Too much rubble. What I mean by this is that the map is not huge, but it's incredibly frustrating to navigate. You are constantly blocked by giant ruins when you move around the former urban areas, forcing you to rely either on the metro system or fast travel. There is also no travel system whatsoever besides the arbitrary fast travel, making the game seem sometimes too big and sometimes too small. Too many locations in the game were also bland and generic IMO. Look at how much more fun the locations is New Vegas is. Giant dinosaur town? Anachronistic roman army camp? Fucking LAS VEGAS!?
GTA: San Andreas: Too big sometimes. This is a tricky one, because if the cities are too small, they don't feel like their real-life counterparts (not that I'd know, because I've never been to any of them). This is mostly a gameplay issue, since you can't save while on a mission, and some missions force you to travel long distances (and do it again if you fail). For free exploraion though, i think San Andreas is almost perfect. It really is huge and wonderful when you just travel around to see the sights.