I would have to agree with your observations. It's that fear of losing control that hinders the experience for people. People who have identified with their ego and mistake it for their true selves seem to have the hardest time with psychedelics which tend to dissolve one's sense of ego. They mistake it for losing their minds and get really frightened.

I have done lots of psychedelics and tried my hardest to purposely use them to destroy my ego and, like weeds, it keeps growing back. But I also have access to another part of myself that lives outside the ego so it was never frightening to me to witness the ego perish and I could just retreat to that other place and watch it like watching a movie, knowing it would return once I came down.

When you let go of the feeling of having to be in control and let the mind wander in this new pool of information it feels like an oasis to take refuge in and is quite enjoyable. I'm giggling now thinking about it.