Only two games ALMOST drove me to destroy my controller. One is Ninja Gaiden on the original Xbox. The fight with Alma is probably the most frustrating thing I've ever done in gaming. I never finished it either. Got really really really close one time (like, one blow away from beating her), and then I died.. The savepoint before that fight is terrible. You have to swim quite a bit, fight three of the most anoying ninja's (they throw explosive shurikens at you) and then fight Alma. The worst thing is that you have to beat those three ninja's without losing any life or you'll have a really tough time against Alma.
The other game is Mirror's Edge. The time trails are horrible! Trying to get three stars on a trail can be practically impossible because you can't make any mistakes, and if you keep making one tiny mistake near the end, when you already did the rest perfect... well it's frustrating.