Lately, I've been thinking Orson Welles might have had the right idea with War Of The Worlds, a story where the aliens are ultimately wiped out because of Earth germs.
Just about every planet out there will have the potential to have evolved completely differently than Earth. The significance of this being of such severity that our immune systems might be so absolutely incompatible that they don't even have a chance to learn how to defend the body from alien microbes (and possibly vice versa). By landing on an alien planet, you'd basically be trying to rush billions of years of immune systems that evolved alongside pathogens in an instant - that's not going to happen; it's suicide.
This means it's not the life-bearing worlds we should be excited about, from a colonization perspective; colonizing a dead, sterile world is much, much less work than trying to move into one filled with species of microbes that our immune system is billions of years out of alignment of knowing what to do with.
So, if we live in such a busy universe, as many scientists speculate, here's you answer as to why we haven't been invaded yet: because life-bearing planets are nothing more than a massive bio-hazard to aliens. It's nasty enough that they wouldn't even want a planet in the same solar system, if they could help it!
The funny thing being that this is a real knee to the groin for most modern science fiction stories, which usually rush to have the protagonists sleeping with aliens just to enhance the sale potential of the novel.