Hello there, it's really great to see people asking serious questions about volutaryism and not just rambling about it.

First things first:

I'll not pretend i know all the answers, we are all searching for then together and there are some great solutions to a justice system and private security, but the main point of most ancaps (at least the jusnaturalists) is that the state is immoral because it violates all the natural rights, and rely on the use of force to obligate people to participate on it without any legitmacy (read this article from the mises institute for a more in depth explanation https://mises.org/library/argumentation-ethics-and-liberty-concise-guide ).

The actual answer to your question

One of the reasons why the state fails on protecting its people is because it has no real incentive to do so, what happens to the state if it fails to deliver a quality security service ? Absolutely nothing, you cant stop paying it like any other private voluntarily hired service. If you have a stateless society you would not have taxes, than most services would be cheaper and most people would have more money to spend, with a free market of private courts and security companies there would probably have a lot of competition which generates lower prices and increased quality, you would probably cheaper and better security available.

You can ask "what about the poor people that can't pay to prosecute a criminal or pay for private security ?" This was a problem solved on medieval iceland. If you had your propriety violated by an agressor and couldnt pay for its prossecution you could easily sell your right to prossecute the criminal to someone that will buy it prossecute him, sue him and than get himself the indemnification/atornement this would also create another market of criminal hunters and increase even more the security.

Why would people respect the law if you can't initiate force ?
The only two laws would basically be the NAP (non agression principle) and to keep your word/honor your contracts, you can make sure that people would honor their contracts nowadays by using electronic smart contracts like the ones you have on ethereum that activate themselves automatically when the conditions are met, and if someone violates the NAP, you have the right to defend your property, if someone initiates the use of force they are not under the NAP anymore and you can use violence against them, and with an armed population it would be a little harder for criminals to attack people. If someone receives a lawsuit for violating a contract and dont answer it the justice will not take any requests from him anymore and he would not be protected, there would also be an incentive for the different private courts to have the same list of blacklisted individuals and etc...

Fell free to ask me anything i havent covered, correct me or my grammar and sorry for my clumsy english

Best regards,
Lucas