What Red Faction should have done:
1) Capitalised on what made the first game so distinctive and actually used their Geo Mod technology in number 2, rather than alienating fans with a generic run and gun with a few destructible walls.
2) Included Geo Mod as a major feature in subsequent games, rather than just getting carried away with the physics of building destruction.
3) Don't try to have to large a budget, especially with so many other competing shooters out there: get your gimmick/feature, big it up, encourage community support through ease of modding and application of custom maps, then once you've got a loyal following and your IP is well known follow it up with a decent successor that retains the feature of the first, builds on it and maybe even introduces new ways of interacting with it, again without bulking up the budget as much as possible.
Geo mod was THE gimmick of the first game, made it distinctive and kept the name persisting. Lose that, like they did in the second game, and you've got just another shooter with a physics engine, a la HL2 or Crysis (though Red Faction 2 didn't even have that - it was just pure generic shooter).
In this day and age every studio seems to be complaining about not making the return that they want to on their product; maybe that's a subtle sign that they're spending too much, hmm? Rather than try an make the most sales, make the most PROFIT, and a good game made cheaply that subsequently sells well makes more profit than a good game that cost a bomb to make and still only sells well.