For me, the fire temple was the hardest (friggin' roll-jumps). That, and the annoying vulnerabilities of the enemies in the forest temple PO'd me BIG TIME. The easiest I'd have to say would be (on the first play) shadow temple, and on the second playthrough, the spirit temple.