upvoting does not affect your reputation in a bad way. it does not take down form your actual number. What helps your number grow is people liking your posts and reesteeming them . because when they re esteem it.. there's a higher chance others will see it.