I think it's a positive thing when someone downvotes a comment spammer who is extracting rewards and ensures those rewards instead go to people adding value. Scammers can extract a lot of value (this is just one example) and if the community doesn't work to prevent that, more scammers will come until it becomes a serious systemic risk that destroys the system.

The rewards pool is, in a way, a shared collaborative commons. If we don't protect it, it will be abused. Votes are how we come to consensus on how that works.

That said, I agree, people using downvotes to harm others may actually increase.