If the downvote was used as it was intended, then calling it a flag would be wrong.
However, the downvote is sometimes used by whales to obliterate a minnows rep, by constantly down voting comments by the minnow, often on the minnows own post and comments which usually don't even have 1 cent! The only purpose of the flags is to destroy a minnow for revenge and adjusting rewards doesn't even feature in the whale's reasoning.

For example, the way @berniesanders flagged @Skeptic into oblivion. Yes Skeptic made a post that annoyed Bernie and Bernie had a right to flag that post to remove the rewards but did he have to then flag skeptic's comments until his rep was down to -6? The first hit was a downvote but the rest were flags.
Maybe there should be separate down votes which affect rep and payout and only certain people, such as yourself, should have the power to destroy rep.
Edit: Ironically, Bernie's rep was flagged into oblivion while I was commenting to him about his treatment of skeptic and I think that that was wrong too, even though he did get a taste of his own medicine.