I think that the apology and polite explanation of the reason for the downvote is what makes all the difference. Most of the time, the downvotes I come across have no explanation behind them. In fact, the ones I've copped have been from accounts which just seem to be one a downvote rampage and pretty random. I don't think they have a reason for their downvotes, other than that they are upset about something with the system.