I usually flag them and write them a comment saying why I did and if they comment back apologizing I will upvote their apology but leave the flag. I think the flag (downvote) is way under-utilized right now because everyone's afraid of it. But it is an import part of helping to distribute the reward pool effectively and self-governing ourselves.