Even from survival perspective I think it make sense for an synthetic AI to keep humanity alive in order to have a backup for itself. If we imagine some sci-fi scenario, then it make more sense for it to move itself to the Moon or Mars, and keep Earth as a reserve planet where biological species can in case of catastrophy recreate AI again. Killing humanity or keeping only animals would be very time consuming because evolution might take million years and also no guaranteed to be replicated (Because we might have altered a lot of resources that were low hanging fruits, that are now require advanced knowledge to mine)
It doesn't need to keep living humans, it just needs enough information to rebuild them from scratch (DNA synthesis + artificial wombs). This also has the advantage that it can install any cultural beliefs it likes without pre-existing culture getting in the way.
If a rogue AI could self replicate and became space-faring, humans would pose literally no threat to them, and they would also never run out of resources.
If you don't need to breath oxygen, the atmosphere of Earth makes for a pretty terrible industrial base. Oxidation is a constant struggle to many processes.
Having natural extremes of temperature, cold sink and hot sink, with no atmosphere would be the ideal. E.g. Saturn's moon Iapetus would be a good AI outpost.
If an AI squashes any life form or intelligence other than its own, then it also has to fear its own reproduction, or autonomous clones of itself. It's a pretty bad strategy game theoretically speaking.
>If an AI squashes any life form or intelligence other than its own, then it also has to fear its own reproduction, or autonomous clones of itself. It's a pretty bad strategy game theoretically speaking.
Any AI smart enough to be an existential threat is smart enough to formally prove its clones are loyal and construct them with negligibly low probability of any failure that could break this assurance.
Gödel's Incompleteness Theorem applies to arbitrary programs. The AI only has to prove results about specific programs, and it can follow whatever restrictions it needs to make such proofs possible.
The physical world means bit flips are inevitable, but error correcting codes exist. You can apply as much error correction as you like and reduce the probability of failure to an arbitrarily low value.
But it can't verify that for generally intelligent copies of itself.
There is no way for it to formally verify large multi-agent dynamical systems.
It will still need a set of agreed upon behavioral protocols to keep things stable.
Grey goo or paperclip optimizers are real dangers, but super intelligence necessitates these protocols even more than lesser intelligences due to the high leverage and rapid evolution of the systems.
