I'll treat someone to whom I'm delegating the same as anyone else, I may downvote them as I downvote other self voters. They also won't be a repeat customer.

Also, I don't mind a little self upvoting done responsibly. It's only blatant milking the system that I have a real problem with.