When you vote and see that the figure has become smaller, it does not mean that you have reduced it. The decrease occurs when the token rate decreases... automatically. Your vote simply updates the actual payout figure.
Don't think about it and vote calmly.