Downvotes are not affected at all. Neither is their concealing effect. Not even the reputation reduction effect is affected - it is simply limited up to 5 points per individual doing the flagging so as to prevent the "thor-of-hammer" effect where a new entrant can be destroyed by 10 flags of a small player or 1 flag by a large player.