I suppose it may also help if we explain what we mean when we say 'math oriented' and simplified, so here are some examples:
In 4.0, they have you lay out 'tiles' (basically paper cut outs) of all the abilities that you can do in front of you. Each of them explain how often they can be use and what kind of action that they use. When ever you use an ability, it tells you exactly what you have to roll and roll against (for example, a melee attack would be roll a 20-sided die, and add 1/2 you class level plus you Str Mod, and you would roll against your opponent's AC (armor class)) and anytime you used up an ability, they just had you turn your tile upside-down.
In 3.5, you could have a ton of options for combat, and casters tended to get over-powered quick, one round of combat could easily take 20 min (whereas 4.0 might be 5, both depend on whether each player knows what they want to do) Some of the maneuvers were also a bit complicated, for example, one of the most complex maneuvers was called a grapple, which was a four step process, as follows:
1. Your opponent now can attack you, if they hit, you fail your grapple
2. Roll a melee touch attack (an attack that is easier to hit with then most melee attacks), you fail if it misses
3. Roll an opposed grapple check, if your opponents total is more then yours, the grapple fails.
4. If you win, you have them grappled, but there will be more checks later to see how long it lasts
There's a ton more rules for it, and I hope I didn't confuse you with what I said.
Again though, if you like the sound of 3.5, but don't like it's complexity, I would definitely suggest trying Pathfinder, since again, I feel it adds and simplifies everything to a nice middle ground of fun, simplicity, and letting everything be broken equally (take that casters!)

For an example of simplicity, most of the 'fancy complicated' 3.5 rules are now just one simple roll. Remember grappling I just mentioned, it's on roll you make using different modifiers then a normal attack (called a Combat Maneuver Roll) against a special defense that your enemy has (called Combat Maneuver Defense)
Just make sure that whatever you and your friends decide, that you remember one thing: role-playing and having a good time is what will make your gaming experience a blast, and that patience is your best friend. I've played many TTRPGs and have noticed that there will always be moments when you will kind of sit there (usually in combat).
Also, make sure that you get the 'core' rulebooks for whichever version you decide to play. They are the Player's Handbook, the Dungeon Master's Guide (Pathfinder's Player's Handbook is the Dungeon Master's Guide as well though), and the Monster Manual (or Bestiary in Pathfinder). Also make sure you get at least one set of dice, which means 1d20, 1d12, 2d10, 1d8, 1d6, and 1d4 (dx being x-sided die).
And if it helps whoever is the Dungeon Master for your group (i.e. the guy who controls everything but the players) all of these different rule sets have pre-made campaigns.
Hope this helps and that I didn't confuse you, and if you'd like more details, feel free to ask.