Well, luckily the retaliation comes from few bad apples only. Most bigger accounts are reasonable enough not to engage in such behavior, because it negatively affects their reputation. I have downvoted a bunch, and only retaliation came from haejin and some small user whose vote barely affected my rewards.