RE: Robots Will Kill Us: AI and Human Extinction
There will be a brief phase where the first to acquire super AI will be the most capable competitors in the world. Nation States: US, China, Russia and some large corporations Google, etc. The competitive nature will ensure the Bad Actor condition you mentioned - at least to each respective out-group. That is going to be the phase with the biggest existential threat by destruction.
That phase won't last long - 5-10 years and what survives it will move on to a collusive symbiosis with controlled competition serving a feedback balancing role. This phase will have the greatest existential threat by evolution. There will be such capacity and incentive to evolve that mankind as we know it is unlikely to exist past that. This phase will roughly be equivalent to the Misaligned Interest condition you mentioned.
The good news is that we get to survive in the same sense that the apes that evolved into Homo Sapiens survived.
What we consider to be the concept of 'Time' is about to get very interesting, and that is probably more of a concern than the AIs driving that.
It's so interesting. I love thinking about time in relation to how much work an AI can get done. Can you explain what collusive symbiosis is?
It is defined partly by its relation to competition. Competition is a zero-sum-game where the winner takes all.
Collusion is a form of cooperation that functionally redraws the group boundary around former competitors unifying goals and claiming all of the competed resource without paying the cost of conflict.
Symbiosis is a form of cooperation that arises when resource specialization is possible and each group produces a resource for the other.
A feature of competition is that if one party of a competitive system is eliminated, the remaining parties gain.
A feature of symbiosis is that if one party of a symbiotic system is eliminated, the remaining parties lose.
Symbiosis is systemically more stable than competition.
I expect that with respect to AI, our most valuable resource will be our quality of consciousness and a possible symbiosis may be formed with AI where we provide 'why|meaning' as a resource in trade for How, What from the AI. There doesn't appear to be enough overlap between our existence substrate and and that of AIs for worthwhile competition. It would make about as much sense as a cow being in competition with the grass in it's pasture. Also note that the cow has an anti-incentive to destroy the grass, because it consumes it. We are the environmental substrate that AI is coming into existence within. I suspect accidental irrelevance is more of a threat that might cause AI to accidentally destroy us.
Yes, in a winner taker all market there would be planned specialization as you are talking about.
To comment on your idea with providing why/meaning to AI, AI wouldn't need any why/meaning assuming it has built in values. By values, I mean a reason behind action. If an AI is acting without us forcing it to act, it must have a goal and values to create that goal. So why would an AI depend on us for meaning?
It is true, a chain of sub-goals pretty cleanly connects how to why and the difference is primarily that why resides higher up the goal chain in an area where it is customary to use words like intent and identity. Consciousness is not well understood or defined. I was inarticulately suggesting that the value we may represent to the AI will likely originate from some aspect of consciousness. Most of the other things we have to offer appear like they will rapidly be insignificant.
Good point. It is the only thing that AI may not have.
Awesome discussion!