Nothing is impossible if there is a will. The solution: give eveyone a private blacklist. If a downvote troll is on the private blacklist, then he is banned from any activity related to your own account. In principle, the mute function would have to be extended.