I couldn't read the whole article, but just from the part I could read:
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.
> an incentive to make their technology seem transformative and powerful
I have no frame of reference to process this.
Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3]
There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.
We are going to spend the vast majority of our future lives without the intelligence crown.
In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas.
Other modalities are progressing very quickly.
There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.
But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.
We are not in the same galaxy as "ready". What would that look like?
Agree with most of this, it's well articulated and captures how we react to change.
However - 'Humans species perhaps 300,000 years, we are essentially the same' is an enormous misconception. That fact that we lack gross anatomical changes during this period, ignores everything we now know about punctuated equilibrium and rapid evolution. It's highly probably we've had an enormous number of evolved psychological changes during the last few hundred, and even tens of thousands of years. Changes that relate to our capacity to live in large groups, adapt to urban environments, resist disease and so on. We know that's the case simply because acute pandemics become epidemics through herd immunity, and through the acquisition of lactose tolerance etc.
It seems highly unlikely that adaptations stop there. Altering the environment (in the last 10K years that means the built environment) alters the environment of evolutionary adaptiveness. It seems likely that we've essentially domesticated out much of our propensity for violence and increased our capacity for mood regulation.
Obviously it's incredibly tricky to pair these specific behavioural changes to genetic changes -> protein synthesis -> behaviour. Bearing in mind though we're only 20 years out from the first study to link allele variant to behaviour (the COMT Val/Met polymorphism), and the potential controversy around such research, this shouldn't be surprising.
Good point. The rate of cognitive improvements would be slow, imperceptible even over many generations, and virtually non-existent over the less than 100 years of digital automation.
It would be interesting to know how much we changed cognitively since our species bifurcated, and also since civilization scaled up social density.
Relative to the discussion, AI tech is progressing in time frames that are a fraction of a human generation. So we are at a complete standstill in that context.
You're applying broad, long-term, and high-level reasoning to motivations that are more likely short-term and driven by simplistic incentives.
Begging for regulation (and input into precisely what that regulation is) relieves competition, consolidating dependency among the established players, and spreads a very effective fear narrative that what these companies are doing will change everything and therefore they are worthy of mind-boggling capex and valuations.
We have to quit seeing humanist visionaries where the evidence shows businessmen with access to the best strategy and PR counsel around.
If AI is now ascending an economic learning curve from:
1. Extremely useful (Claude Code & Waymo now)
2. Doing ~everything we do (AGI & Optimus in a few years? 10?)
3. RSI (?)
4. Being smarter than any living person at every intellectual task (?)
5. Being smarter than the best-organized aggregate of all humans (10-100 years?)
...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?
Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.
Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.
Here is an issue I think about often, but I am not quite sure how to put it into words.
We have many extremely smart people in various fields. Executives, politicians, and society generally ignore them and do whatever they want. I don't believe that lack of access to intelligence is our problem. How is "free" intelligence going to improve this?
I don't just mean climate, but business planning, health, risk assessment, everything.
There are lots of incentives for politicians and executives (and anyone else holding the levers of power) to ignore information, intelligence, and advice. I think you're right to be skeptical that "free" intelligence is going to improve anything without first addressing the incentives of the people holding power.
AI will not help by improving extremely smart people. AI will help dumb people and dumb processes with "free" expert-level intelligence. Anyone could still ignore intelligence and make ignorant decisions. But the default mode would be highly intelligent informed decision.
Example with health - a patient can read blood test results with Opus and get very good results for "free". This is far away from helping extremely smart people, still it improves society from the ground up.
Yes, this is along the lines of what i have been thinking. Why would anybody listen to a smart machine when someone they feel emotional attachment to make a counter argument? This arguments always assume that people are very rational and will always take the action that will favor them but people routinely do stupid stuff all the time and will defend their right to do those.
Yeah, it's really bad out here man. There is a huge epidemic of even highly intelligent people being manipulated into doing completely irrational things.
"Free AI" will not fight existing knowledge or strong opinions. It will spread to empty spaces. Example with blood test — there is no podcaster there and will not be.
No, and we have wars no one in their sane mind would even imagine 20 years ago, like that of Russia vs Ukraine.
See, it's easy to speculate how there are less wars when you live in a place which haven't seen war for decades or cenuries, but it's a complete game changer when it's 150 km from where you live, and it's not just some regular war, but a long play intentional meatgrinding AI drone debugging polygon.
> it's easy to speculate how there are less wars when you live in a place which haven't seen war for decades or cenuries, but it's a complete game changer when it's 150 km from where you live...
I can appreciate how having skin in the game makes it feel different, but that's like saying, "I know crime isn't down, because somebody broke into my car last week." That's faulty on several levels.
By many--most, even--measures, wars since WWII are massively down globally. I don't expect that to hold as the US strategically disentangles itself from the globe, though.
Many analysts of very sane mind imagined exactly that war 20 years ago, or longer.
The geopolitical fact is that Russia lacks strategic depth, which has bitten it badly multiple times in the past. From their view, that's something to be remedied to prevent future occurrences.
For Russia to gain strategic depth, there are a few lines which it needs to control. There are ~6-10 gaps, depending on how you count them, and on what net importance/counter-productiveness you assign to some of the more marginal and fraught ones. Holding any is better than holding none.
The big surprise is that Putin left Ukraine so late, when many of his best-trained ex-Soviet personnel had already died. Had he done it even just a few years earlier, the outcome would have likely been very different. The only analysts who thought it wouldn't happen thought so precisely because he left it too late.
Regardless, the best time for Europe to get serious about this war was before it started, unfortunately, and there was ample warning.
> No invention was able to steer the psychopaths away from waging wars.
Nukes and MAD did a pretty good job of that.
Your model, that psychopathy is the necessary precondition to war, is a popular one, but it's not grounded in reality or history very well at all. It's a harmful view when broadly held, on balance.
Nations have interests, and nations sometimes go to war to pursue the geopolitical imperatives they rationally believe serve their interests. Computer-controlled drones notwithstanding, there's nothing new under the sun. The unprecedented (and likely transitory) period of peace and prosperity during the Cold War was the anomaly, and we're now reverting back to the mean.
> Being smarter than the best-organized aggregate of all humans
Isn't there pretty much a consensus that committees and institutions are not all that smart?
I think you're confusing the categories of "intelligence" and power. Institutions are powerful. The smartest AI is still just a tool without the infrastructure to turn that into real world effects and someone to direct it.
It seems you have faith that this is inevitable and unavoidable. I get it, even rationalists succumb to religious thinking eventually. We're only human after all.
I'm genuinely baffled each time I encounter this notion that the power is dumb.
It's as if power was a kind of demigod Heracles who paves its way with sheer brute force alone.
But in reality it consists of very mortal and feeble creatures of meat and bone.
How in the world can people assume dumb those who possess all the means and rights for violence by controlling and manipulating myriads of other creatures of meat and bone, is totally beyond me.
Power and intelligence are not completely unrelated. Are government officials, politicians, CEOs, judges, etc, usually above average intelligence? Probably. Do the institutions they work within make intelligent decisions? Sometimes.
The fallacy is assuming intelligence automatically equates to power and influence. University professors and doctors have very little power in their institutions even though they may often be more intelligent than their bosses.
A worse fallacy is assuming intelligence creates better political outcomes (in the broad sense of politics as contested collective decision-making).
I don't know many people who would willingly trade their freedom to make a bad decision for the enforcement of the "right" decision by a superior intelligence.
> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.
I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.