I think you're right on about avoiding downvotes as much as possible. The downvote button should be moved elsewhere (harder to click accidently), the icon should be something like a trash can, and perhaps there should even be a confirmation dialog.