Everyone can downvote freely, and no one should agree or disagree to that person's reasons. Now if you are receiving the downvote, and you know you've done nothing bad, than that sucks. But nothing can prevent abuse, without actually removing your freedom to downvote.