Removing completely flags is not suitable. Perhaps replace for a new mechanism. Must have some mechanism to solve spam, auto vote farms, abuses, to down rate content, etc.. Some of this situations could be handled automatically by the code others like rate content is very difficult without the help of a human brain.

Spam, auto votes could be easily handled by software.

Abuse or rating content need other way to be solved. Perhaps separate the content's ratings from the payout. When you downnvote you just lowered the rating of the content and this rating with the votes, weight, etc. give the final payout.