As far as which sport has tougher athletes, I'd say either hockey or rugby. But I haven't played either one so I can't be certain. I just go by what I see on TV or the internet.
As far as the toughest sport to play, I'm throwing my hat into the "baseball" pool. I know I'm gonna get some crap for this, but seriously, baseball is one hella difficult sport to master. It's the only sport where if you fail 2/3 of the time you're in the Hall of Fame. What other sport can say that?
When hitting, a baseball player must make split second decisions about:
1. Determining the speed of the pitch
2. Determining which way, if any, the ball is breaking
3. If it's a ball or a strike
4. What part of the field do I want to hit this pitch to
5. When do I start my swing
Most of the time these decisions have to be made in under a quarter of a second.
Sure, you don't necessarily have to be a great athlete to be a great baseball player, but the skill set needed to play baseball at the highest level beats out any other sport IMO.