The counter argument is probably that self-improving AI would need to be self-aware.
If we command it, why? Why can't we just tell the non self-aware AI to go and improve its own design? It may even deduce that the design is its own, but it may be so composed such that it simply doesn't care.
The advantage of a self-aware AI is that it can come up with goals that you didn't think of. This is the quintessence of the "double edged sword." Humans are already quite good at this, however. As William Gibson wrote, "The Street finds its own uses for technology."
Eurisko has already demonstrated that non self-aware AI can arrive at of ways to satisfy goals you never thought of. (So have Bayesian spam filters.) This is already a powerful tool that we haven't exploited even halfway as well as we might.
If we command it, why? Why can't we just tell the non self-aware AI to go and improve its own design? It may even deduce that the design is its own, but it may be so composed such that it simply doesn't care.
The advantage of a self-aware AI is that it can come up with goals that you didn't think of. This is the quintessence of the "double edged sword." Humans are already quite good at this, however. As William Gibson wrote, "The Street finds its own uses for technology."
Eurisko has already demonstrated that non self-aware AI can arrive at of ways to satisfy goals you never thought of. (So have Bayesian spam filters.) This is already a powerful tool that we haven't exploited even halfway as well as we might.