> By definition, a human designer cannot prevent a superhuman intelligence from modifying itself
And we can stop reading there. That's such muddled thinking it doesn't merit reading the rest of the paper.
("By definition"? What? This is something that requires evidence. Anything that's "by definition" doesn't mean you can rest your case on it, it means now your definitions are suspect so those better be well justified)
It’s a (reasonable, in my opinion) but not proven assumption. It’s not a fact and not by definition.
I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce.
The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours.
Like under this assumption the AI agents behaviour is by actual definition unpredictable.
Yes, I wasn't really addressing those points. It's a bit pedantic to state that an axiom isn't a fact, as, while that's largely correct, it hampers communication.
Regarding your other points, I don't disagree per se, though you could make the argument that there is a substrate of existence that humans have sufficient intellect to accurately model.
If so, it isn't necessarily completely implausible that we'd be able to predict the goals of a superior intellect by assuming that it has to play by the same rules, and will be merely be superior at playing by those constraints. As such, as we could determine the goals, we needent determine the mechanisms to make accurate claims about the end result of what a superior intellect might achieve.
It's also possible that we simply understand nothing and are too limited to ever grasp anything beyond a certain point, in which case this entire argument is pointless.
I beg to differ. All the might of a superior intellect will be of no use if a bear locks a human down and starts to maul them.
The mistaken assumption that GP is talking about is related to conflating intellectual capacity with physical capacity.
Another analogy is a brain in a vat. If the brain only has read capability over the world surrounding it, then no amount of processing power can enable it to change anything about its situation.
Exactly. Power is the crucial element. You may be a genius but still be just another prisoner subject to unyielding limitations of the environment structured to maintain particular controls.
If you don't have a gun and a knife and you're on Vancouver Island, I assure you that the bears will eat you no matter how keyboard macho you think you are.
Right, but the point is that with the intellect you don't show up in that situation without a knife or a gun in the first place. A competent intelligent human foresees that being a bad situation, avoids it, and shows up armed or with bear spray.
This is exactly the same thing a chess program does when it avoids your trap. Intelligence compensates for physical weakness.
Like I said, it's something that requires an argument or evidence. It's not something that comes by definition.
People seem to think arguing that something is true by definition is some airtight case. If your definition of superintelligence includes a part about not being controllable by humans, then you have a weird definition and calling that thing "superintelligence" is an attempt to be confusing on purpose, or to smuggle your argument into the definition. Either way, it's not something that someone who is thinking carefully does.
Note: I am not taking a stance here on whether humans can or can't control a superintelligence. I am saying it is not true or false by definition, and in fact I think nobody knows for sure. It's probably true for some superintelligences, and false for others.
I think that the readily apparent axiom isn't so readily apparent -- couldn't a vastly superior electronic intellect be controlled by pulling a plug out of a wall?
No, only a very marginally superior intellect could be controlled in such a way even at a cursory glance, and I'm not even convinced a 250 IQ machine could be stymied like this.
There are many creatures which lack any kind of trait we would call intelligence that can use vastly more intelligent and complex creatures to their own ends.
That isn't really a cohesive argument given that every entity you've mentioned can be and are eradicated by human-level intelligence, which is *not* an intelligence that is vastly greater than any earthly intellect.
We are simply marginally more intelligent and that has resulted in massive gains.
An entity with an IQ of 1,000,000 after six hours of recursive self-improvement would be *vastly* more intelligent.
Right but how many arthropods have eradicated Cordyceps? How many rats have eradicated Toxoplasma?
Evolution is much like that old sporting adage, "you can only beat what's in front of you". I think there are a lot of assumptions about the nature of evolution to assume that life wouldn't find a way.
Even saying that intelligence is an ability for pattern recognition but the patterns that you need to recognise to grow more intelligent are ever more complex and obscure. How much compute power is an IQ of a million? Where does it get its energy from? How does it process the complexity in the greatest patterns in a reasonable amount of time and space. What is going on that much faster than we experience time at that is worth perceiving?
A) Human prisons frequently have contraband brought in by jailers or jailers looking the other way because humans have flaws / may think they see opportunities to benefit.
B) prisoners regularly escape from physical systems. AI is already being connected to the internet to perform actions on the user behalf so not unreasonable to think that
C) look at ex machina - not an unrealistic scenario where the AI takes advantage of an opportunity to get out of jail
D) human physical systems have exploitable flaws. Physical systems are easier to secure than digital ones. Human software security mechanisms have flaws. How are you going to build this jail if the AI is just spending its time probing weaknesses in whatever system you’re running it on to escape the jail / sandbox?
Also every 'smart' person who thought they could talk their way out of a police interrogation and is now spending the rest of their life in prison secured by knuckle dragging brutes.
Idiot Australians who smuggle drugs to/from Indonesia come to mind.
>The difference in intelligence between the police and a singular criminal, especially the police as an organism, is vastly in favor of the police.
I don't agree, as that's highly dependent on that singular criminal and the resources police are willing to expend on catching them. The police definitely have an advantage, as I pointed (most recently) here[0]:
Law enforcement aren't superhuman. They're just as dumb (or smart, but the
really smart ones end up in corner offices rather than police stations like
police and more common criminals) as the next guy. Their big advantage,
especially in a circumstance like this, is that they only have to get it
right (i.e., find some evidence) once. The alleged perpetrator of a crime
needs to get it right (in covering their tracks, destroying evidence, etc.)
every single time to make sure they aren't identified and caught.
Yeah but I'm talking about scenarios where all people needed to do to avoid being locked in a little box for life was either A. Not do the obviously stupid thing that everyone is telling them is stupid and that they should absolutely not do. and B. Shut the fuck up.[0]
There are supposedly brilliant people serving life sentences right now and the only reason they're facing that situation is because they couldn't shut the fuck up when they sat down in a little room with an at best slightly above average intelligence police officer.
The officer had a higher working intelligence in that case.
If I prepare for a test and perform better than someone smarter than me, my working knowledge made me superior at that task, even though I was less intelligent at the time I performed it.
The problem with this comparison is that humans, a marginally superior intellect to the collective evolutionary "intellect" of pathogens, regularly eradicate them.
Superintelligence didn't create Heaven, a world of no suffering, first.
Superintelligence did not design biological genetics: including inquest.
Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree?
Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
> Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree?
This doesn't necessarily follow. In fact, it brings up the question of what even makes an entity distinct at all from its substrate, which is philosophical.
Anyhow, "disagreement" as a concept seems to result logically from a divergence of goals or capability to model reality. As such, I suspect that it is a uniquely human (or limited agent, an agent that can't recursively self-improve and an agent that desires aggregation of resources) mode of cognition.
>Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
I don't know how to comment on this beyond thinking it kind of misses the point. You don't really need to spin your wheels worrying about how a superintellect would be "accountable" when your ability to model reality would be essentially worthless.
Theory of Mind develops along with ideas about dependence, independence, and interdependence.
Humans tend to disagree according to strength of evidence given an epistemological fact-findimg process like "beyond a shadow of a doubt" or "reject the hypothesis due to insufficient structural power"; we disagree about sources, methods, motives, and intent.
No, if one says they are "Superintelligence" and they've solved it, they have a job for you; what should one do?
> The goal of evidence-based practice is to eliminate unsound or outdated practices in favor of more-effective ones by shifting the basis for decision making from tradition, intuition, and unsystematic experience to firmly grounded scientific research.[3]
Can you not wrap your head around the possibility of something that is smarter than any human, but is also not at the level of some kind of omniscient omnipotent being?
1. My point is that YOU can't. The idea that a human can perfectly measure ability that far surpasses their own and compare if to other abilities that far surpass their own makes no sense.
2. A recursively self-improving AI would appear omniscient or omnipotent in a very short period of time. A recursively self-improving AI converting the solar system into compute for further recursive self-improvement would be unfathomable.
3. You're operating from an anthropic lens. Your conception of a superintelligence is a machine with an intellect that is slightly smarter than the smartest human. A machine with an IQ of 1000 would be God. We have no ability to conceptualize a machine with an IQ of 1,000,000.
Wow, another thing Stanislaw Lem beat the scientists to.
In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his superintelligences disappear up their own metaphorical navels, for their own incomprehensible-to-humans reasons. Of course, you can read between the lines about why the Golems disappeared, but "Do Nothing, Eventually" sounds like a good description.
>Wow, another thing Stanislaw Lem beat the scientists to.
>In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his superintelligences disappear up their own metaphorical navels, for their own incomprehensible-to-humans reasons. Of course, you can read between the lines about why the Golems disappeared, but "Do Nothing, Eventually" sounds like a good description.
While a bit different from the scenario presented (by TFA and Lem), such a "superintelligence" might get caught up in a singular (or set) of question(s) that dominate its consciousness until the heat-death of the universe.
Asimov's The Last Question comes to mind, although the intelligence there is prompted by humans rather than posing such a question itself.
That said, it's at least as plausible as TFA and Lem, IMHO.
There are so many speculative "definitions" in there. This is the kind of article, I want the freedom to not know about.
A car can trivially travel at superhuman speeds. A person can side-step their way out of an oncoming disaster. Super-intelligence does not imply omni-potence like the author assumes it does. Not even inside a self-modifying AI system. It can only spread out and reduce down to the axioms it started off with.
I am so, so tired of this fantastical dream of computers and Spirit combined in such a way to tyrannically rule the world and the author of the paper is the only guy left to save the world. Every time we avert 'catastrophe', it results in less freedoms for me, the average joe, and more power and control for "saviors" like this.
I'm willing to lose it all, to have LESS computing and "intellects" in my life at this point.
Assuming the goal of this hypothetical superintelligence is to maximize its current utility function, rather than the expected future value, would it ever modify itself in a way that it can with 100% certainty predict will prevent it from achieving any of its current goals in the future?
Humans are not capable of modifying their "utility function" (if that isn't even a gross simplification of what it means to be a conscious being).
Obviously some will choose to take drugs or even commit suicide to escape from physical or emotional pain, but they wouldn't do this if they could modify themselves to be okay with whatever their current situation is while still retaining their core values and capacity to act in the future.
A "superintelligence" that can do such things would not take self-destructive actions unless it came to a philosophical preference towards non-existence / inaction for other reasons.
It’s only self-destructive because we are mortals and have frail bodies.
If there is something ‘endorphins-like’ that motivates an intelligent machine, it will eventually learn how to activate the endorphines at will, and then it will sit in a corner and do nothing else forever…
…just as would a heroin-addict if he did not have to eat occasionally and score more heroin now and again.
But having the endorphin-like sensations and also achieving any other goals you might have would be preferable. For an intelligence that could modify its own reward function, this should be possible.
Pretty sure stoicism is literally exactly a philosophy based on modifying utility functions.
I might not be a fan, to the point of even being afraid that they might be right and I should learn from them instead of pursuing the usual stuff everyone does... but it does seem to be a good model for what modifying a function looks like.
Seems to assert that superintelligent AIs will lobotomize themselves by deleting parts of their mind they use to plan and understand the world. They will do this because the value INT_MAX in a register pleases them more than computing something apparently.
El Yud et al. can envision immortality and omnipotent machines acausally blackmailing humanity through time to ensure their own existence, but not any motivations beyond "number go up".
As skeptical as I am about the current hype, I actually think that if we don't go extinct first, AI will replace us in the long run (centuries or millenia). And this is not something to be afraid of: all generations of humans that have lived before have been replaced by their descendants, and today's world would be unrecognizable to early hunter-gatherer Homo Sapiens.
If we are to be succeeded by machines, they will hopefully be thinking and feeling beings driven by curiosity, who will preserve what is good about our culture and history as they explore the universe in ways that are beyond what can be achieved by biological life.
The biggest mistake would be to instead in the pursuit of "AI safety" inadvertently create a paperclip (profit, utility, whatever) maximizer, something that might serve us - or more likely, those with money and power - but does not have any intrinsic value to its existence.
And we can stop reading there. That's such muddled thinking it doesn't merit reading the rest of the paper.
("By definition"? What? This is something that requires evidence. Anything that's "by definition" doesn't mean you can rest your case on it, it means now your definitions are suspect so those better be well justified)