I have always been against automatic vote, if you really build relationships with other users, it implies the reading of their materials and live comments on the subject.

But, I understand that it's hard to find free time for this (every day follow updates of friends and participate in discussions). And auto-vote in a sense is an opportunity to say Hello, to express respect for your relationship. Accordingly, if you withdrew from somebody list you have lost the interest of this person to you. So your actions in this case are absolutely correct.