Yes. Stake works on redirecting the pool in both directions, it isn't a right to payout. So if people want to downvote what someone upvotes, all is good, like @flysky is downvoting comments I receive as he seems to be trying to influence people to not comment - unsuccessfully. He downvotes my posts I guess to influence me to stop posting, again unsuccessfully. There is more to influence the system than stake alone.