Yes.

Since downvote is the exact opposite of upvote it would make sense to use these consistently. Flagging is a different action than upvoting, it seems now, while in fact a downvote and an upvote are the same, just expressing a different feeling.

Implementing this would only make sense if it's also implemented visually: by showing that it's 'the opposite of the same action' by placing upvote/downvote next to each other - instead if in two different locations, which suggests it's a totally different action.