Thanks for the post. I wrote about this, too, in AI: To pause or not to pause. First and foremost, IMO, it's not possible to pause. Even if some companies agree to stop, that just increases the incentive for others to continue. There's no way to enforce a worldwide pause. So, it almost doesn't matter if we should or not. I don't think we can. Pandora's box has been opened, irreversibly.

You made the same point, here:

Even if these nations pledge to stop AI development publically, they will just continue doing it secretly under the table. No one will miss the opportunity of 1000x productivity, and no one will trust the other to do so. It's not utopia after all.

and here:

History teaches us that technology keeps going up no matter what skeptics, naysayers, and critics say. I have a feeling that the same applies to AI. For better or worse, I believe it will continue to develop and advance even faster than ever before. Time doesn't turn back and that's why we got here in the first place...

Beyond that, since we know about the vast array of benefits from AI including health, safety, prosperity, and more, the risk of harm is not sufficiently clear to justify foregoing all of those benefits.

Finally, people imagine AI systems outcompeting humans for resources, but that's not how it would work. Humans will always be assisted by state of the art technologies, so the AIs would actually have to outcompete AI-augmented humans. I think that humans with technology assistance will always be able to outcompete AIs alone.