*sigh* Define selling-out please? Actually don't. Just forget it. I'm sick of people throwing that term around. Slayer ARE mainstream, they're the biggest fucking thrash band on the planet. Just about everyone has heard of them, even those not into the genre. Same with Pantera, you know how many metal-fans hearts were broken when Dimebag died? A LOT. Just because something is mainstream doesn't mean it's not good, as we can see by those two examples.
However OT best rock band currently not in the mainstream.... I'd go with... Rolo Tomassi, although they're steadily approaching it.