Yes, it is wrong, but for some reason, it often comes naturally, especially when we don't like someone, as you pointed out.

Maybe it comes naturally because in every person there is at least something that we don't like, so we get a little satisfaction because of their attributes that we don't like, and we feel empathy because of their attributes that we do like.