As an Unrealscript programmer, I really, really don't think it has as much to do with the tech being inaccessible and unwieldy as it does with game developers and fans alike being shortsighted and ignorant. If anything the tools have improved to the point that anything we could have done in a year back in 2000, we can do in a few weeks now--and much more reliably. Check out Kismet if you don't believe me. This is a very small example, but a door used to take ten minutes of scripting and trial and error with numbers to make work properly in old Quake engine games, now you just link up some nodes and re-position the door visually.
Go back and actually play Deus Ex for a while, and consider just how wildly different its level design is from 99% of all first-person games, Half-Life included. The sad fact is that nobody imitated it, because it's very difficult for designers to wrap their heads around exploration-driven environments. A level designer's job is to impose a sense of structure, and it's hard to understand how to have both structure and freedom at the same time. It CAN be done, and it's not a technical problem at all; it's an organizational problem; an issue of design.
As long as developers continue to see exploration and narrative as being mutually exclusive, that's the problem we're going to have. I would like to point out that this is a state of mind brought on by Half-Life, which is one of the most railroaded and controlled games ever and began the trend of using scripted events, which demand railroading. The games that have good exploration value don't have these (Arkham Asylum, Deus Ex, Crysis 1, inFamous) while games that feel like they need to hold your hand do (Uncharted, FF13, Call of Duty, Crysis 2, Bioshock to a lesser extent). Such games end up depending on the spectacle of their scripted events over design knowhow because actual design knowhow is difficult even for a really fantastic game designer to possess, while spectacle is often a matter of sheer brute force.