Have you noticed that your follow is going to decrease for a while?
I think this is because you are not giving upvote their comments.