Thanks! At this point rarely anyone notices I'm taking their photo. But the thing is I mostly shoot blindly without raising the camera and looking through the viewfinder cause I like to leave the scene as it is and not interrupt people. And my Fuji has silent shutter.
Before that, while I was shooting with the dslr, I was uncomfortable getting this close cause everyone notices the big camera and thinks you are pro. Like this, even if you raise the camera to eye level they think you are a tourist and just snapping photos. It helps when you act like one. :D

But I honestly never had anyone complaining if they see me taking the photo of them. And if you are clear about your intention, if you like capturing the life of the street or whatever, then you will be just fine. :)