If you un-upvote you don't get the vote power back. So you just lose that vote. It surprised me when someone told me that so I tested it to confirm it.

Also, how are you working out which position you are in? The drop down list doesn't reflect timing as far as I know. There's no way for someone to bump you in the timing of votes. It could be they're voting while you're reading the article and you haven't refreshed the page when you vote?