I've never actually been in a fight myself, but I know what you mean, and agree with you.
I imagine it's something to do with human empathy, or the likes. But seeing a fight gives you a shot of adrenaline, probably because of basic instincts of 'oh crap, I may have to fight in a min..' so...