I think there are certain degrees of "caring" about what other people think. If the person "judging" me is a good, smart person - someone that is a benefit to society; I care what they think. On the other side, if the person judging me is a classless human with terrible people skills, I don't give it the time of day in my mind. You have to consider the source that is making these judgements. But yes, in the end, you are the judge of yourself that should count the most.