I think just like you said, one must avoid negative people if one is aiming to build up his self confidence because negative people have a way of breathing negativity to ones self confidence affecting it at the end of the day negatively