Never, underwear is usually comfortable and interchangable, whereas trousers might be made of rough material that might be irratatable to your...err you know..
Plus you don't want hairs or unshaken fluids going onto the bed or damping your trousers, it probably isnt nice to wake up with hair in your mouth...now just think if it was a hair from...down there. D: