To me, the biggest annoyance are the explosives when combined with Critical Hits.
I'm not a big fan of games with grenades and explosives, because the bottom line is that players quickly master them then usually spam that as a primary weapon (Battlefield, Call of Duty, etc.). Limiting the explosives to just two classes, the Soldier and the Demoman limits this, since the Demoman has a considerable arc and slow firing weapon, and the soldier has a fairly slow firing weapon and a small clip, it isn't too bad. But that's when the other part of the equation comes to play, the Critical Hits.
Criticals allow these classes to spam unholy amounts and just bank on the splash of a critical to do the job. The splash on a critical rocket isn't uncommon to see kill two or three players, and the Demoman can rain his weapon better due to a higher clip size, so he's just as dangerous in the right hands.
Together, I find this just very frustrating to deal with. But I may be a bit biased, since my primary class is Medic, meaning most of my kills are with the melee Bonesaw or the Syringe-Gun (which takes a bit of practice to get right at longer distances, crazy arc on that weapon). As a fix, it would be nice if the splash on the critical hits of these weapons was significantly reduced. I don't mind a direct hit killing me, but the destructive potential of even a miss of these weapons is a bit much.
The class balance seems pretty good, although I agree that the Pyro seems to get the short stick. He is a close range specialist, with really no other abilities, but at close range a decent soldier can usually beat one 1 vs 1 by just shooting the ground. I only have really good luck getting kills when I get up behind an enemy, which then makes me wonder why I'm playing Pyro and not Spy.
Critical Hits as a whole I think are alright. They add another wildcard to the game. It doesn't add luck, because in all games with humans there is luck and chance involved, since humans themselves introduce it, or it can occur as a result of outside influences. Criticals allow that Engineer defending his equipment with a shotgun a chance against that surprise Soldier, or the Sniper a chance against that Scout. Aside from the explosives, I don't think a fix is needed.
The Respawn timers seem absolutely needed as a result of level design, which I frankly don't mind. I usually play Call of Duty, Search and Destroy, wonderfully fun, and if you mess up you have three to five minutes to think about what went wrong. A twenty second respawn timer is fairly forgiving, but also without it many of the levels are terribly broken (try playing 2fort without respawn timers, unless the other team is unbelievably bad, it's impossible to get out of their basement with the Intelligence). The only real "fix" for that would be reworking the levels, spawn areas, and their distances from the objectives, which seems like a bit much.
Overall, I really do enjoy the game. I usually just play the Medic because I like helping the team, but other roles can be fun to goof around with. I'm really looking forward to the changes Valve has said to be planning as to the classes and new features, as well as the additional maps and mode to come.