I keep my phone silent when I do not want to be disturbed, and later I explain to the caller; if we starting worrying what the other person would think if we don't pick up the call, life would become very difficult :)
I too think about what other people would say if I don't pick up the call, and that is why I keep my phone silent, and later, call back.