It really doesn't even need to be all of us; as long as enough people take responsibility for ensuring justice, or do what they can do (I think here of a man in a wheelchair who might have to yell for help rather than confront a mugger), there would be no need for central authority.

But there are some problems humanity needs to defeat within itself.

1. We do not have a unified sense of right and wrong

centralized authority wants to put people into cages for smoking a plant taking some other drug

It is not the government that called for Prohibition; it was a mass movement of people that put more of a moral hazard on drinking than on the intrusion of government into personal rights. This set the stage for the "War on Drugs" fiasco. Of course, rent-seekers within the government have taken full advantage of these situations to increases their own power.

2 We are kind of cowards.

We have given up our personal responsibility to be eternally vigilant

We pass on our own power to the state because we don't really want to be the one that confronts the mugger. The State and/or Society can socialize people to be more aggressive or more passive, but there is only a certain amount of people who are naturally confrontational. Most people don't want to do it, and it takes aggression to counter injustice.

These are not insurmountable problems. We need to push for a personally accountable and personally responsible morality that recognizes the moral duty to act when necessary.

All free men fight, all fighting men are free

Good analysis.