Depends who you're speaking to. I would skip any variant of "I wasn't a fan of their work" when speaking to someone who was close to the deceased unless they specifically asked me, because it's just unnecessary and tactless when they may be feeling quite sensitive. Generally however, it's just true, and it's no insult to happen to prefer other things. Consequently it shouldn't really be a problem to just say to people who didn't know the deceased personally.
Second element is "I'm sorry he's dead". If you're not sorry and so don't feel you want to say that as it's a bit dishonest, it can be skipped over, straight to something like "Best wishes to his loved ones", which is showing sympathy and respect enough for their loss.
If you're not sorry someone is dead and their loved ones are so awful you don't even want to express sympathy for them, you're probably talking about someone despicable enough that you shouldn't be showing remorse at all. However, if for some reason it is absolutely necessary to comment, you can always go to a further remove with a generality such as "I heard about that. It's always a sad time for people to lose a close relative and friend". This is a superficial, factual recognition of loss sufficient to sound respectful, but has no personal commitment of opinion, feeling or respect to the deceased and their loved ones at all. Of course, those more socially savvy will notice the phrasing and the implication that you actually couldn't care less.