For me, the main thing is the instrumentals. I'll readily listen to, say, Joe Satriani's stuff (which is entirely instrumental) because the guitar work there is phenomenal.
Second comes the vocal style (not the lyrics themselves). If a song has a vocal style (example: screamo and the group of 'X-core' genres in general) that I dislike, but otherwise has damn good music, I'll listen to it and enjoy it for that.
Third is the lyrics themselves. I listen to music for the music, rather than the message. I don't speak a word of german, but I think Rammstein are brilliant. I can't understand a word of melodic death metal, but there's some fantastic guitar-work in there.
Here's an example of descending order: My Chemical Romance. The music itself isn't actually half bad. There's a fair few decent riffs and some technically good work going on. However, I'm not hugely partial to the 'american pop-rock' vocal style, and the actual lyrics themselves are melodramatic to the point of silliness (and this is coming from a guy who likes power metal).