Speaking as a fencer who has seen both weapons(or their modern day sport equivalents) in action, I'd have to give the edge to the rapier. The skill, blade quality, and circumstances of the duel have been heavily discussed, so I'll just assume that both combatants are having an unarmored gentleman's duel, both have equal skill with their respective weapons, and neither weapon will break.
The first difference is reach, in which the rapier comes out the victor, which means that the samurai first has to remove the threat of the blade before getting in close to strike. However, this is something the samurai will have trouble with, as the fencer has been trained to keep his blade from being beat out of the way. The samurai also has a significant disadvantage if he tries to hold the blade, as his guard is not designed to do so, which makes it much easier for the fencer to free his blade and counterattack as he moves out of reach. Finally, the samurai's stance and small blade guard leaves his hands far more vulnerable than the fencer's, as one hand is kept behind him and the other is protected by a guard designed to take hits from heavier blades, and the fencer is trained to use a circular parry to redirect much of the force of a stroke as well as holding it, giving him a significant opening to exploit as he moves out of the samurai's range. If the samurai is skilled enough to protect his body, the fencer could simply attack his hands with little worry about a counterattack fast enough to reach him, and eventually the samurai's hands would be injured to the point that he can't fight effectively.
The second difference is speed. While the katana is very fast in the hands of a skilled user, and can thrust with surprising effectiveness, it's wielded with two hands, which makes its thrusts slower, and its cuts are slower than the thrusts of a rapier, which require no wind up. If the samurai could close distance, however, he would have a far better chance of coming out victorious, as his blade is shorter and made for cutting. The rapier is quite sharp, but it lacks the weight and curve to land a lethal cut anywhere save perhaps the neck, particularly up close where his blade has little leverage.
The third difference is in footwork. Here, too, the fencer has the advantage. The side facing stance allows both for a smaller target and for more extension in the lunge, further adding to the fencers reach advantage. With my observations of both styles of movement, I would have to say that fencing is the faster of the two, at least with regards to opening and closing distance, making it very difficult for the samurai to close the distance and attack.
Keep in mind that this is based on my personal observations and experiences with both styles of fighting, and do not consider it a definitive comparison.