Rapier. My reasoning is simple: Let's presume a historical encounter. This would mean that both participants are plucked out of their respective timelines to square off against each other, which would also mean that they would have no knowledge of the other or their fighting style. Let's also presume that both participants are wearing no armor, only using a single sword and they are masters of their respective school of swordsmanship. Finally, let's presume that this is a fair and square duel where factors such as drawing speed and terrain advantages are negated.
Under these parameters, the fight would go like this:
-The styles they use will be unfamiliar to each other, as it should be, so neither of them would bum-rush the other. However, while katanas are wielded similarly to European longswords and broadswords, something the rapier user would logically have a counter against, rapiers and the related fighting style would be completely unfamiliar in Japan, and thus the samurai would have no immediate counter against it. This means that the musketeer will have the initial advantage in the fight.
-Using this initial advantage and mixing it with his technique's greater flexibility and range, the musketer would be able to deliver the first strike, and even if it's not fatal, the wound would definitely hamper his opponent and thus give him even further advantage.
-After this the combat would be more balanced, as after the first strike the samurai would now have a better idea about the style of his opponent, thus helping his parry his attacks more efficiently.
-In the end the outcome would be the same: the wounded samurai would slowly bleed out while the musketeer would keep his distance and he would deliver the killing blow once his opponent is anemic enough. The only way the samurai could come out of it on top would be if he would throw caution out the window and bumrush the musketeer early in the fight while hoping that he wouldn't receive a fatal wound in response.
Under the aforementioned circumstances I would say 6/10 times the musketeer would win, 3/10 times they would inflict debilitating and/or fatal wounds on each other early in the fight and in the remaining 1/10 times the samurai would be able to parry the first strike and then overwhelm his opponent and cut him down before he could recover his balance. Overall I would still put my money on the guy with the rapier.
Now, on a bit of a tangent, I think the OP should have really clarified that this discussion is less about the actual weapons than the fighting styles associated with said weapons. It just breaks my heart to see so many people only focus on how the katana is made of pig iron or how the rapier would break and whatnot, which is completely irrelevant at best and factually untrue at worst.
In melee combat skill is everything, and doubly so when you introduce slashy/stabby bits of sharpened metal into the equation. Even a legendary katana would be just as effective as a wooden club in the hands of an amateur the same way a master fencer could kill his opponent just as easily with a fire poker as with a masterwork rapier. It's all about the man, so please stop arguing about the metal.
Speaking of which, someone around here was arguing that the katana got too much hate in this thread. For this I would say it would be more accurate to say that katanas are just receiving the backlash they deserve after being so ridiculously hyped up. You know, karma and stuff.
But again, why are katanas hyped on the first place? My best guesses would be these:
-World War 2: As it was already brought up before, there are many stories of Japanese soldiers with katanas cutting gun-barrels and whatnot. Now, why would the American soldiers make this up, you might ask?
There could be two reasons: there might have been an actual anecdote flowing around that got blown out of proportion (say, a soldier used his gun to block the sword of a Japanese soldier rushing against him in the trench and his gun broke/fell apart from the encounter), or more likely, propaganda, likely on both sides.
On the Japanese side they naturally hyped up their mass-produced swords as the best thing since lukewarm water (since that's what armies do to all their equipment, no matter how old or inefficient; just look at North Korea), and on the American side as well, for psychological reasons. To put it bluntly, it was so that the soldiers would feel more threatened and would have less reservations when it came to shooting guys only armed with cheap replica swords when it came to assaulting Japanese bunkers once they ran out of ammo. After all, those blades are dangerous! They can cut through a gun barrel and sever you in half, you know!
(Also, Mythbusters. Their methodology might not have been the best when they tested this, but I mostly agree with their findings: a katana could, at
best, cut into an overheated machine-gun barrel under ideal circumstances, nothing more.)
-Anime: Probably the biggest cause, the anime and manga industry just never runs out of ideas when it comes to over-hyping katanas, often in the silliest and most implausible ways possible.
However, for the starting point of the katana-hype, we would have to look further back. It actually has its roots in the old samurai movies, practically the only genre besides some chinese wuxia stories where weapons were often as, if not more important than the characters. Take the weapon and battle-centric skeleton of those stories, add high-schoolers and other stereotypical anime tropes and you have got practically every shounen anime ever.
In these the katana has a set of tropes associated with it that makes it awesome beyond reality, like the ability to cut steel, other swords, to cut a person in half so fast they would not even realize they were cut until a few seconds later, the implausibly fast iai cuts and, of course, the whole bushido-mentality 95% of katana-wielders embody (except the villains, but there is a good chance those guys would either use some exotic weapon, european swords or just plain old guns anyway).
These are all over the top stuff, but remember, in animation being more awesome beats implausibility, and it's arguable that the viewers would slowly associate the katanas themselves with the awesome scenes courtesy of the simple fact that there are usually no other types of swords in these, or if they are, they are either some super-special-awesome-mega-blade the the protagonist uses or they are wielded by the antagonists (and since they are wielded by the antagonists, they are never as cool as the ones that are wielded by the heroes).
-Back to the samurai movies for a bit: you see, until the late 80s, those movies were practically the only ones where swordplay was a cool and impressive part of the film and the only movies where you could see elaborate swordfights. They were to katanas what Hong Kong martial arts flicks were to kung fu.
Then came the exploitation era of filmmaking, and then we got the well-toned Americal action heroes using kung-fu and katanas to beat up their opponents. Why were they using katanas? Well, because they were fighting ninjas, of course.
Finally, even in our modern era, the trope has taken root so hard that katanas keep showing up even when they make no goddamn sense. Like say, in the Highlander series, both McLeods are using katanas even though it makes no sense, with them being, you know, highlanders. Or, since we are on a gaming site, I might as well bring up things like how katanas are all over D&D, how a demon from hell makes a katana for one of his sons in the Devil May Cry series or how Mass Effect 3 has space-ninjas with space-katanas that can one-hit-kill you.
At this point katanas are just thrust into the hands of the characters just because of the tropes associated with them, because of tradition and, well...
-They just look awesome. Not only that, they look distinct and aesthetically pleasing to the eye. Smooth curves, a pronounced edge, polished and shiny with a recognizable grip pattern... Katanas are at the perfect intersection of looking good in the hands of practically anyone and having a unique shape that people can instantly recognize, unlike the myriad different types of swords Europeans developed. Also, they look delicate, and it is a well known psychological effect that people associate delicacy with skill and speed. Put a claymore into the hand of a guy and he will look like a brute, put a katana in his hand and he will look like a skilled warrior.
So yeah, this post turned out waaaaaaay longer than I planned. Sorry.
