Depends on the music and the people. If it's a metal, or some sort of heavy, chaotic band that's good to mosh to, then the only factor that's left is the people.
Sometimes I'll go to a concert with a band and venue that tends to attract a ton of asshole bodybuilders and jocks that just want to beat people up in the mosh pit, so I end up just standing around and listening to the music. And even with that, people will still be assholes. The last one I went to, me and some friends were standing up at the front of the stage, and some guy just came by and started slamming into us and throwing punches until a security guard stopped him. Or at another show some of my friends went to, they did a silly, fun dance in the middle of the mosh pit, but it wasn't "manly" enough, so some asshole slammed them into the ground.
At least in my experience, those kinds of shows haven't been the norm for me. When I went to see Between the Buried and me, the crowd was way cooler, and the mosh pit consisted of people pushing and shoving each other around, but not to really hurt each other. When you get knocked down people will help you back up, and if don't want to be part of it, then you will be left alone. There's crowd surfing and other cool stuff, not everybody participating is some hulk. Those kinds of shows are the best.
I really hate it when people mosh overly aggressively to actually try to hurt people for pretty much any band.