I think without 'free' downvotes nobody will seriously try to stop any kind of abuse (why should he pay and not others?).

I agreee that 'free' flags are used mainly by whales because all others fear retaliation. That's why I am in favour to implement arbitors/committees of users to protect people from retaliation and flgas given just for personal reasons.