I am thinking that one good solution would be looking into system of moderation & metamoderation on Slashdot, and perhaps similar systems elsewhere. Basicly, everyone can moderate from time to time (upvote/downvote), and those up/downvotes come with tags. They liken it to jury duty. Meta moderation is 2nd round, were people moderate how the moderation was applied to random comments.

Perhaps the ideas from a system like that could be used with some modification?