Don't forget that when you notice many votes appearing in a row, some of those might be in a curation trail following the first voter.

But its true that if not a bigger user votes, it won't raise the value too much. But with time and balanced distribution this will get divided more evenly eventually.