I would say unfollow users that you are not reading anyway. Following someone would actually mean that you care about their content. If the person has a low UA, then following them actually is helping them get attention from other people. If you unfollow, it's because their content no longer interest you.

Also, following a strong UA doesn't do anymore good than following a weaker account.