I had a friend in high school who was sort of a weird kid and a stranger randomly punched him in the face but maybe that was something different. I think what you are referring to is how people are just rude and mean to people who perceive as weak, my wife is a petite woman and people always make rude comments to her who would never say shit to me, a large man. A lot of it is your intentions too, if you are constantly thinking about this and projecting it into the universe that is what you will get back.
I see your reasoning, but if you read my story you will see it was more orchestrated than unconnected rude and mean people seeing weakness in me and acting on it. It was and is much more.
I was reading something about a fellow conducting research where he would take a criminal for a ride up the street and have him identify who he would victimize and multiple criminals would always independently choose the same people. People they perceived as weak for some reason.