Well usually I'd wear either nothing or a swim bikini.
I wear boxers only if they're short and they have the kind of waist like pajama pants.
I traditionally used to wear briefs but recently stopped because I just had it. And not because the reasons you'd think. They are always cut wrong and don't conform right to your body because they make for guys with stuff the size of eggplants it seems (and I get the smallest size possible btw), and the damn "hole" thing up front is bothersome because it's constantly rubbing against stuff down there, and most of all I hate the elastic band because it always squeezes the waist!
I only like stuff that has small, short waist band. Unfortunately I don't have anything like that except the swim bikinis.