I have read and tend to believe that people care more about their stuff than yours. For example, they hear what you say, maybe. Then go back to thinking of their own stuff. Basically what you are saying in your post.

Some people are really hung up on what others think and sometimes those that worry like that were trained to do so by others that were taught to be that way.

The old adage: Haters gonna hate.
haha! Not sure I added anything to your post .. it is early here.