Fallout 3 all the enemies they added in Broken Steel, damn those guys have hit points through the roof and/or a lot of armor.
Fallout: New Vegas, Cazadores, probably because of my very first couple of experiences with them they raped me so frikken hard, also because they are ridiculously hard to snipe, those buggers won't stay still for more then a fraction of a second, now that i'm high level i breeze through them though i still hate them.
also in both games i loathe bloatflies and radroaches, definately not because of any danger they pose , i'm just really annoyed that they clutter up my "radar" getting me on edge every time for nothing.
cliff racers in morrowind, annoying as hell, after a certain point no longer a real danger but they just keep throwing themselves at me wasting my time and resources.
goblins in oblivion, again the resources required to kill them are just not justified when the payout for killing them is so low. i don't mind spending a crazy amount of time sneak killing everything i encounter as long as the reward is good enough, the loot they drop however doesn't deserve my characters time investment.
also not a monster but the entire dream sequence in dragon age origins is annoying, the first time it was a nice puzzle to solve, everytime after that though it's just annoying it takes too long and it's mostly just running around collecting the special powers to be able to progress.
pretty much any pokemon game, zubat and any other worthless pokemon that dares cross my path and refuses to let me run away, i usually stock up on the repellant so i don't run into pokemon in the parts that only spawn annoying ones.
basically any monster or enemy in a game where the effort doesn't match up to the reward given annoys me, also any enemy's where it isn't skill that determines whether you win or lose but luck does, i hate monsters that have a broken power that insta kills you or something like that which it does 50% of the time that's just lazy game design to artificially prolong the game time of players.
so yeah that's basically it.
enemies you can't avoid/run away from.
enemies without proper rewards.
enemies with broken powers.