Majora's mask is the strangest Zelda game there is.
But I wouldn't consider it difficult. (I played Master quest not too long ago though, so perhaps that's influencing my judgement of it.) Just... Different. - If you don't understand the way it works, you'll have a hard time with it.
It also has a few rough edges suggesting it was rushed.
Then again, since it's closest relative, Ocarina of Time took 3 years to make, perhaps that's understandable.
On the whole, it's a very impressive game. I can understand why it doesn't have more mainstream appeal though.
Then again, that doesn't stop me wanting someone to pick up the concept it started. (I love time travel.)