There would still be abuse. We would still have upvote bots that cost about 2-3X the reward pool draw which compounds with regular use, we would still have the comment farmers, and we would still have the misused down-votes. But there would be less of the reward pool pie for them to take if people did what you say.