I thought it too, but what if assholes create lots of account and start downvote on other users at the same time upvote themselves? Or what if a low-rep user move SP into other new accounts and upvote each other?
I think it can be the chicken-and-egg problem.