It would seem to me that putting a downvote button next to the upvote button takes away the 'flagging' idea and then it would be similar to a reddit style voting system (which is a language that people are quite familiar with).

thats the way it originally was (a reddit style down chevron). The "flag" exists in the UI only. The idea was to discourage people from using it arbitrarily, but it had the unintended effect of of causing far more buthurtedness.

The voter identifications aren't the problem (and yeah, its easy to see it on the blockchain)

The problem is that if the aggregate total of votes is displayed, it doesnt draw everyones attention to it.

100 upvotes and 1 downvote just shows as 99 total votes. But with the big red flag easily identifiable on the side, it shows as 100 upvotes and some guy telling you to fuck yourself.