I'm too new to be sure but I do believe to downvote you must first upvote and then remove that vote to create a downvote

I'm uncertain if one can do that more than once...
I don't think I'll be one to use this feature either...

If it be egregious enough I wouldn't hesitate to flag
but the rule of thumb is live and let live here
or so I believe...at this time