Another idea: since the UI of the site is still in early development, one really important improvement that would go a long way to combating this is being able to see a followed user or hash tag. Once this is done it will be very easy to organize down voting.