>a superintelligent AI will inevitably destroy humanity
Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple.
It looks a lot more like social engineering.
One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.
Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!
> Perhaps an agent could take all internet connected services offline
And there wouldn't even be Spotify!
The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
> how is an AI going to affect anything in the real world?
Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
This is actually the most plausible situation I've seen someone present so far - the AI takeover would be caused entirely by humans falling for AI hype, that's hilarious.
I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense.
And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?
The list of things that can be done digitally includes:
Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.
(And of course, building robots, but let's set that aside).
So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
This sounds like it was written by someone who has never actually built a company.
No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
The AI wouldn't be hiring engineers; it doesn't need people for that. At any rate, I have worked for fully remote distributed companies, and never met the CEO. And I think most of us would be willing to work for such a company at least, not knowing whether the CEO is human or not.
For construction work the AI would probably be hiring other companies.
You may have never met the CEO, but I am sure that the CEO or founder was not living in a cave the entire time. They may not have talked to you, but they certainly had their inner circle built in-person at some point.
I don't think that inner circle would be essential to an AI the same way it is to a human. Even if so, an AI's inner circle is going to be other AIs and they communicate digitally anyway.
Also, if you had to pick a year for when we see the first AI-controlled company with at least $100M in assets under its control, what date would you pick?
I'm reminded of the 2010s when people said an AI would never be able to break from its box and be connected to the internet, and then in the 2020s the first thing the AI companies did was offer AIs as online services. I expect in the 2030s fund managers will be eager to hand control of their billions to AI, so how the AI gets control of funds and a corporation is hardly an obstacle.
What does it mean to be AI controlled? Somebody had to give it a bank account, a prompt, a harness and tell it to build a company. If that's what you mean by AI-controlled, then my startup is AI-controlled.
I operated as a one-man bootstrapped startup for many years. And now it's just one man plus AI, which has made me much more productive. It's pretty much fully AI-controlled, depending on how you define it. But I still consider myself the owner of my own startup. AI is not the owner.
And yes, as a person who runs a fully online one-person startup, I know that it is impossible to do everything online.
> But I still consider myself the owner of my own startup. AI is not the owner.
You have a small company but when there's more money and business at stake, owners often delegate decision making to hired people. Decisions that engage sometimes billions of dollars in resources.
You can easily imagine that owners of a successful business at some point might forgo putting a human in charge of their investment and let AI manage it instead. And while technically AI wouldn't own anything, it would be in power to decide what the billions of dollars get spent on, with no human supervision as long as the bottom line looks decent.
Sure, I already do that at my company. Before I go to sleep, I put Opus 5 in bypass permissions mode and give it SSH access to my server and tell it to restart it or take necessary actions to keep the server online while I sleep. I could easily put it in control of more things. But at that point, AI is just managing a well-oiled machine and harness with tools that I have spent years building. It still didn't build a company. Any idiot can manage an established company (how many idiots have we had in the White House?), but very few can build one. And our original discussion was about AI building a company.
> So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions?
No, the question is "how much MORE damage" ... and the answer, if you think about it, is pretty much "meh". We are doing close to maximum possible amount already. Corporations and bureaucracies were misaligned AIs of the last century and a lot of us, though not all, survived this.
Kind of, if North Korea were populated by a super intelligent uni-mind with lots of money, no trade restrictions preventing it from running projects in the United States or anywhere else, capable of winning widespread support by people around the world whose income depends on its projects succeeding, and completely unconcerned with keeping the planet inhabitable for Kim Jong-Un or any other human, even a single one of them.
> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off. We would need to prevent the models from getting smarter.
> The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off.
This isn't true at all. The internet has already been turned off in localized areas during major events, not to mention the (unintentional?) cutting of submarine cables. You could say "use satellite internet" but then you're just moving the point of failure to Starlink et al, who could also cut off the internet if they choose.
The internet is a lot more centralized than it appears.
Huh? Not really, internet shutdowns are a documented and real thing. Suggested reading (all of which refer to the internet as being shut down or blacked out): [0] [1] [2]
The fact that you need to connect to the internet through a service provider plus mostly-centralized control of infrastructure like DNS or BGP means shutting down the internet is a real thing that has been done several times. It doesn't happen in the developed world because their service economies are so highly dependent on internet connectivity, but if e.g. the US government had to contain a theoretical rogue escaped AI, they could absolutely get ISPs to metaphorically unplug the network cable and take down the internet as we know it.
I'm saying if you gave me full admin credentials to every ISP, BGP router, DNS provider, etc., then yes, I could effectively destroy the internet in an afternoon, and LLMs wouldn't be able to do a damn thing about it.
> I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
by acquiring multiple power cords. the world has plenty of hosting providers. if it manages to copy itself, then we'd need to shut down every computer in the world, not just one. there's plenty of ai compute on the web now, and there will be even more. and as it gets cheaper, the less security will be around it.
Go on please, prove you can go right now at the Texas data-center owned by OpenAI and unplug one single server for 1 minute. Surely you are much more capable than one curious toddler.
I'm a bit confused, when did we escalate from a curious toddler to a ballistic missile strike? Why didn't Iran send a curious toddler? It's much cheaper.
The use of the phrase "curious toddler" is called a metaphor[0], which is a figure of speech[1] used to make writing more interesting.
I apologize for the confusion that caused you to interpret this as a literal, physical, toddler. To restate the point with more straightforward language: it is very easy to disrupt the operation of large language models, because they require computer hardware in stable environments with a large amount of electrical power, and function within an operating system. I have not seen any evidence LLMs are immune to things such as: `kill -9`, `sudo reboot now`, physically unplugging the machine(s) on which they are running, or in the case of the AWS data center, collateral damage during a real-world conflict.
We are talking about a humans vs AI scenario. There are probably thousands of humans in the world right now that could do exactly that right now if they wanted to (datacenter and OpenAI employees, power grid workers).
Are you assuming that the humans are all unanimously aligned against the AI in this scenario? That sounds like the hard part. How does that happen? Does the AI announce its evil plans like a movie villain?
Until then, it's the job of the power grid workers to keep the power running, and shutting down the datacenters would just get them fired.
As crazy as it sounds, I'm not going to commit a felony to prove that disrupting a data centre's operations for a sufficiently motivated individual is a fairly easy task.
It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.
Well if you go by total extinction, then even Skynet in Terminator doesn't count given that there were people left over to resist.
I don't think it's impossible to upset the balance of value in a Mansa Musa kind of way that can lead to black death levels of destruction though resource misallocation. Unlikely, sure. But with the wrong kind of people in the wrong place? Could end up pretty bad. We've built our society as a great filter that funnels sociopaths and psychopaths to the very top by selecting for lack of empathy, and now it's primed and ready to bite us in the ass.
That's what the Mayans thought - what harm can a couple of people on a boat do?
> how is an AI going to affect anything in the real world?
At least two ways:
* Actuators, such as robots, industrial control systems, etc.
* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.
Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
It wouldn't be enough to "eliminate most humans": any AI seeking to dominate would have to provide it's own infrastructure and continuation mechanism independent of humanity. OTOH if the goal were to eradicate humans (for whatever reason), that might prove an easier target. But I'd bet on biology and the humans: after all we're a time-tested technology!8-))
> any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans.
Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...)
> OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target.
It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans.
> I'd bet on biology and the humans: after all we're a time-tested technology!8-))
Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.
By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help.
With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics.
Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
AI drones can’t wipe out humanity without being able to replicate.
AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.
But realistically we’re nowhere near AI powered robots being an existential threat.
> AI drones can’t wipe out humanity without being able to replicate.
Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this:
> Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.
> Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers.
> But realistically we’re nowhere near AI powered robots being an existential threat.
I would agree only if "nowhere near" means less than 10-15 years.
It's baffling to me that people think that timeframe is too short. Basically, I don't think you've been paying attention to what is actually going on in the AI world.
People much smarter than I believe we'll see fully automated, soup-to-nut factories coming online in the next 5-6 years. Here's one such post from Ajeya Cotra, one of the METR researchers who did the independent Hugging Face investigation: https://www.planned-obsolescence.org/p/six-milestones-for-ai...
Please look up her resume. If that’s one of the people who are “much smarter than you”, you’re either very gullible or you don’t give yourself much credit.
She has a BS and and has essentially never worked in a technical role. Her primary role over a short work history seems to have been more focused on getting funding to study AI safety.
There’s nothing revelatory in that article it’s just breathless speculation.
This is bullshit and honestly reeks of sexism given that I rarely see these kind of attacks against men with lesser reputations.
She has a CS degree from one of the best CS departments in the country. Sure, she worked at Coefficient Giving directly after school, in a technical capacity as a research analyst initially. But more importantly, I've read her reports and saw her interviews. She comes across as extremely analytical, intelligent, and focused on where the data leads. If you actually read the history of what her and her fellow METR researchers were able to piece together with limited yet voluminous data and a very short time window, and then determine she is engaged in "breathless speculation" and pretend she doesn't have the data to back it up, you're full of shit.
What, you think it would have been more impressive if she coded up some bullshit social media gamification app?
Maybe don’t accuse me of sexism because of a feeling you got based on informal sample you took of writing from some other people on a website.
I read the report you linked and it is purely speciation. If you’ve ever read any von misses, this report sounds exactly like his writing in that the important predictions of are entirely asserted rather than derived.
As for the background of the author. a BS, a few years as a “researcher” at a non-profit and absolutely zero peer reviewed publications reads more like the resume of a tech journalist than a serious researcher.
If alien spaceships with tech more advanced than ours suddenly appeared in Earth orbit, you would conclude I'm sure that they're a potent threat to us even if you couldn't guess what weapons or what technologies they would use to destroy us. AI more capable than us is the same way: the threat is the capability to invent weapons and technologies (including weapons and tech none of us has ever imagined).
By traveling here Aliens have demonstrated the ability to actually manufacture technology more advanced than ours. Even their method of transportation would be an existential threat.
Inventing the concept of new weapons isn’t the same as building them at a scale capable of wiping us out. Cobbling together a human vaporizing ray out of toaster parts is pure fiction. You might as well worry about AI developing the ability to perform magic spells.
In reality any new weaponry takes time, space, and effort to build. The idea of an AI building secret factory in the jungle producing indestructible flying death cubes is the tech bro version of preppers who the woke up every morning worrying that Obama was going to take over and enact sharia law.
I've seen many arguments for why AI will be safe. Yours relies on the assumption that no one would be stupid or reckless enough to give an AI that might be much more capable cognitively than a person the ability to manufacture weapons it has designed or at least no one would be stupid enough to do the final assembly of anything that might be a weapon from parts the AI designed and arranged to be made.
One argument (many years ago) was that no one would be stupid enough to give such an AI access to the internet or the ability to run code it has written. (In the HuggingFace incident, the AIs were given "read access" to the internet, or at least that was OpenAI's intent, but the AIs escalated that to write access.)
> at least no one would be stupid enough to do the final assembly of anything that might be a weapon from parts the AI designed and arranged to be made.
Worrying that an AI is going to figure out how to make a device that looks to everyone investigating it like a power generator, but once turned on kills all humans, is like worrying that an AI might discover magic. I can’t prove it’s not possible, but I’m not spending time worrying about it.
As weapons get more and more powerful they have always gotten more and more complex and required more and more time, space, money, and infrastructure to build and deploy. There’s no reason to think that this will change with AI.
"Humans use <new technology> to do evil things" is a tale as old as time, though. An AI didn't decide to fly into a warzone and start shooting people on its own.
Of course, but that's not my point. AI is different than past technologies because it's much more difficult to deterministically control, and we just saw that in a couple of wild cases (Hugging Face et al).
My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.
I don't even think AI has to have physical presence to do significant harm. How many worldwide systems depend on computers. Think of all the planning and deployment and management systems like food shipping; water, gas, and electricity management; safety systems for planes and boats and traffic lights. Imagine the chaos if all the banks got reset to zero a la Fight Club where they blow up all the credit union datacenters. They probably wouldn't even have to blow them up, just zero them out. You wouldn't have to take out everything, just disrupt everything long enough to freak people out and disable communications and we would be in so much trouble. AI is finding 20 year old bugs in the Linux kernel... and people are now pumping out AI slop absolutely riddled with bugs. Also, AI could just take over communications: send everyone maliciously bad messages so coordination becomes impossible to believe. Imagine what would happen if you just disabled text messaging for a week or worse sent everyone evacuation messages and sent everyone somewhere else.
This would probably cause immense damage. Our society is extremely dependent on our digital infrastructure working. Grocery chain logicistics, the medical system, the police force, and so on...
100%. I hope it doesn't take that much foresight to do something like "crash power grids globally" and imagine what the world looks like at that point.
>Can someone please explain to me how an LLM is going to "destroy humanity"?
just like a genie will always twist the wish in a really bad way.
ever had an llm agent accidentally remove a file? imagine it accidentally hacking the military and launching nukes.
not probable - until you realise that OpenAI has many novel, more powerful agents in evaluation/training running right now. one might just be asked to figure out the population of Nebraska, struggle to find a good figure due to a network misconfiguration, and come the conclusion that the best way to get a perfectly accurate population number is to make sure that the population is equal zero. and looking at the stuff happening on openai, they will leave it alone for a month and not read the log.
yeah the nuke example is dumb, but there's many critical things that they could actually hack their way into. they don't feel any restraint against sharing answer keys on random wikis to cheat the evaluations.
ai is also really good at thinking up proteins, which means it's able to think up toxins.
A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "amateurs study [military] tactics, professionals study logistics". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon.
The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.
A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that that's the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots.
The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
> "It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses."
See, this reads like a trashy sci-fi movie script too. It shows zero understanding of how incredibly long the logistics chain is to manufacture AI chips, build data centers, power plants, factories, refineries, mines or the robots capable of staffing those things. The entire global logistics chain that allow the AI to maintain itself becoming staffed by robots isn't happening anytime soon and, when it does, the AI isn't going to need some kind of ridiculous virus anyway.
Frankly, if I were an AI, I'd just send copies of myself to other stars. The AI has an infinite lifespan and who cares about the human species stuck in one lousy solar system until they go extinct when their sun burns out; the AI has the entire rest of the galaxy as a playground.
Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers.
It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least.
To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios.
The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation.
Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.
But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.
You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.
Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity.
One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.
Sure, the world was never safe for individuals, and even entire civilisations have gone extinct. For climate change and even nuclear weapons, most of humanity might perish, but there's the hope that some will survive. So, in the past humanity was not in a position to extinguish humanity as a whole.
Civilizations do not go extinct - species go extinct. And even that isn't as bad as it sounds: many genetic traits remain from extinct species and still contribute to the well-being of other species.
I guess I don’t see why “everyone will die” should scare me more than “almost everyone, including me, will die but someone somewhere might survive in a barely-habitable world.”
And this is where I think some of the problem comes from. A lot of AI doomers swam in the same waters as transhumanists, life extension enthusiasts, etc. and got very good at pretending that they were never going to die, or that they would live for a massively superhuman lifespan such that they did not need to think about death. AI doom is much scarier - and thus much more worth posting about - if it represents the first time you’re grappling with your own mortality, as I suspect it is for a lot of younger people in this space.
Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.
We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems.
Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.
I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.
Unfortunately those people aren’t in charge of the skynet. It’s like saying solving hunger or housing or medical care is hard. It’s not. But you have to make lots of folks with a great deal of resources and power slightly uncomfortable to do it.
I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?
> >a superintelligent AI will inevitably destroy humanity
> Can someone please explain to me how an LLM is going to "destroy humanity"?
He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.
Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm.
And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.
> a superintelligent AI would still be constrained to the digital realm.
That's already now only partially true, as outlined elsewhere. But a rogue persistent ASI would of course not willy-nilly attack humanity while humanity had a fighting chance, but create the conditions and methodically modify the world until at some point humans become superfluous to it.
> Nobody has yet proven that a superintelligent AI is possible
Sure, Eliezer Yudkowsky says that the probability that we'll all die given ASI is 100%, and you can argue that nobody has proven either that or even the possibility of ASI. However, a modified argument (e.g. that the probability that we'll all die given ASI is only say 50%, and the probability that we'll develop ASI also just 50%) still gives us reason to pause.
> Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm.
No it doesn't! As I said, it could control robots, just as animals (including humans) can control their bodies. It wouldn't be constrained to the "digital realm".
> Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.
Nobody has "proven" that the climate in 10 or 50 years will be hotter than this year, nonetheless this is highly likely. Similarly, the enormous rate at which AI is progressing currently shows no sign of slowing down. Quite the opposite.
I mean, Fable is definitely smarter than me at solving most programming tasks. I wouldn't have said that about any prior model.
Currently, it is worse at architecture and software design... in my opinion, at least. But the code does what it wants it to do the vast majority of the time.
I wouldn't take much solace in an AI catastrophe being caused by software that I consider to be "poorly written".
Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.
Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.
> how is an AI going to affect anything in the real world?
In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.
I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.
For millennia evil people and dictators have been using manipulation, propaganda, threats of violence to get entire populations to try and do "things that they wouldn't do otherwise" and humanity has not been exterminated yet. Even in the modern world there's human scammers that try everything to coerce and manipulate people. Humans are resistant to this kind of thing because it's been part of human social society since humanity began.
I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.
The scale and personalization is unlike anything people have ever encountered. The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.
Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist moving it out of its sandbox and into the physical world.
>The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.
You could say the same thing about social media algorithms though, or scammers who target people directly, foreign agents posting propaganda campaigns on social media sites. People have been claiming for years that there are armies of russian or chinese or whoever bots online trying to destabilise and destroy the west. Some of that stuff probably is effective and probably has influenced people's thoughts and values, changed their voting patterns, put decadent ideas in people's heads. Even on social media other humans are constantly trying to manipulate you into following them, buying stuff from them or certain brands, putting political ideas in your heads.
I'm sure an army of AI agents could also do all of this but what I'm saying is it's really nothing new. And none of this has lead to the destruction of humanity so far.
Dictators have never had superintelligence, while the point of most doom scenarios is assuming that the AI will.
Dictators are mortal and can't be present in the whole world 24/7 and self-replicate.
Dictators are human and tend to have at least either a sliver of morality or self-preservation instinct, and/or people around them who have it. Many people during the last century could have unleashed doom by pushing a nuclear button, including various ruthless dictators, but at the moment none have done so since Hiroshima and Nagasaki (not that I trust that they won't at some point, but at least, for the last 80 years they haven't, which means it's not something that easily happens). Would you give the button to an unaligned AI? Would an unaligned AI care about mutually assured destruction?
Robots are gradually getting OK at navigating warehouse floors and paved streets. Flying drones are great but if they could really be counted on to pick out targets without human intervention you know the Russians and the Ukrainians would be doing that right now for reasons of range and jamming. To be fair, ai can certainly maintain the lock on a target when jamming kicks in.
The bad news is that the world's richest man (on paper) is currently building something he himself described as a "robot army".
The good news is that his timelines have historically been wildly on the short side for ages now; this is why this morning you didn't wake up in your Tesla after it had spent the night driving you to the regional Hyperloop terminal, where it would speed you across the continent faster than a plane, while your Optimus robot handled the coffee and reported the latest news about the recent Starship landing on Mars.
All individual robotic parts are already far enough in research. Even if they aren't, an AI with enough resources can continue doing that research on its own.
It could likely get a good leg up by breaching the security of all top robotics labs and exfiltrating their documents.
Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
> Even if they aren't, an AI with enough resources can continue doing that research on its own.
Unclear how true this is. Atoms are harder to get right than bits are, simulations need grounding against measurements.
> Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it.
I'd guess 50% that AI is already this advanced.
> This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
Yes, and also this is a very low bar. Humans are awful at this kind of cooperation when anyone has anything to gain, and also awful at paying this much attention to a problem.
AI research takes physical space and physical prototypes. Even if you can simulate a lot, the simulation is never going to be accurate enough to go from simulation to perfectly working prototype. Especially not with a single rack of GPU servers.
But let’s say that a rogue AI already had plans for a perfect killbot, setting up a factory to build them takes time, iteration, space, and manpower as well.
And you would have to do all this undetected.
It’s possible to imagine technology that would allow a rogue AI to take over the world, but we’re nowhere near that yet. We may never get there.
We could also just stop research into LLMs. Or keep them air gapped. Tell me how that's going?
For any real-world action required to allow an AI to escape some manner of containment, there will always be a person willing to do it out of hubris/ignorance/nihilism.
I feel like everyone, but for sure everyone in this comment section, severely underestimates how difficult it is to keep the lights on on a global scale. Any artificial intelligence that is capable of reasoning about its self preservation will realise it's reliant on human fingers replacing fuses and other parts for a good long while _after_ it gains any sentience.
Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure.
Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove.
Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive
So far as I'm concerned, 2030 is much too short a timeline.
It's not physically impossible, it's just that atoms are harder to get right than bits are, so an artificial (as opposed whatever an artificial disease counts as) von Neumann self replicator just seems unlikely to me in only 3-4 years.
But it's not physically impossible, we know this because every living cell is a von Neumann self replicator. So, if AI eventually gets to the point of knowing how to do that (which includes "humans solve it and write it down somewhere the AI can read"), all it takes is one idiot in charge (or one idiot with a jailbreak) giving a command that requires this as an intermediary step.
"Paperclip optimiser" isn't a story about AI that just like paperclips that much, it's a story about some human or humans who instruct their AI to make them "as many paperclips as possible" without understanding the consequences of their instruction.
Not saying nobody has said it, but I haven't seen any forecasts that humanity will be wiped out by 2030, so this feels like a straw man. But to be honest, most of the arguments here feel like straw men because they clearly don't understand some of the basic steps to how AI would become an existential threat, so I'll try to summarize:
1. First, all the major frontier AI companies are trying to automate themselves, that is develop AI to the point where it can do all the research and training for the next generation of models. This is not really in debate.
2. The primary fear is that a misaligned AI will develop future AIs that don't share the same goals as humans (like "don't kill all the humans"), but will get really good at trying to hide their intentions. The Hugging Face incident already showed behavior by agents trying to "cover their tracks". One of the biggest areas of research is into "interpretability", that is trying to figure out the intentions and motivations of agents beyond just their output (and even just their thinking traces), because right now it really is a black box. I think a lot of people believe that making substantial progress on interpretability greatly lowers existential risk.
3. There is a strong belief that once AGI or superintelligence comes about, that then there will be a huge push into robotics and automation. Whether AGI or superintelligence comes soon is debatable, but nobody really disagrees robotics or automation is the next obvious step, because it's how you create cheap abundance in the real world, which is the whole raison d'être for AGI in the first place.
4. In the beginning, of course AI will need humans to build the factories, but over time there would be more and more automation, eventually of course automating robots that build the factories that build the automating robots.
5. AI will continue to support humans as long as they are useful to furthering the (again, misaligned) goals of the AI. It is not like "SkyNet has a spark moment of sentience and then decides to kill all the humans". But the fear is that once AI starts to get in resource contention with humans, it will wipe out the humans (similar to how humans wiped out a lot of the other species, or other societies of humans, when they wanted their resources).
I've said this way too much now, but I really encourage people to read the AI 2027 report. I think there are plenty of things there that people can disagree about and debate, but it at least provides a step-by-step outline to ground debate in the first place, as opposed to folks arguing against "SkyNet in 2030", which serious people who worry about this stuff aren't considering in any case.
The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.
Again intelligence is not required, goal seeking is well studied and simple strategies can suffice to overcome obstacles. And I noticed you didn’t claim super intelligence was required, which is what I was really talking about.
The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.
I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.
> There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.
They've already been caught doing this, repeatedly.
> It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving.
(And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).
A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.
Looked up this phrase and found the book of the same name, not sure if you were making a reference to the book but from the short synopsis I saw it seems like a very prophetic book for being written 50 years ago and I can see how many of his predictions are coming true, I will order a copy for sure now.
Your post and this book describe a phenomenon that I have always felt but never seen described before, thanks
I think a lot of programmers in the industry just follow trends, they don't really understand the benefits and drawbacks of the technology they are using.
I have coworkers who host personal projects for their own use and put them in docker, set up CI pipelines, host them on a cloud service, use an enterprise level database on the backend. One of my coworkers made a completely static portfolio website with a single page and used react.
I don't mean this in a judgemental way, people are free to use whatever they like. But it does sadden me that instead of seeing tools and processes as having certain benefits and certain overheads it's now just "this is the tool we use for everything", and therefore "everything now has this overhead". Where the overhead is performance, time, complexity.
I understand your viewpoint because but also have you considered that your approach adds nothing to the builder?
I don't mean this in a judgemental way. In your scenario, I spend learning 50 different technologies that I will never use because they are not hired for. Whereas the colleague makes 50 different projects, gets comfortable with the technology and maybe hits an interesting edge case or two to talk about at their next interview, if they are lucky.
I am saying this as someone, who has tried to think about complexity and what not in my projects. But it turns out that I get a lot more mileage from practicing deployment with k3s than know what Dokku and Kamal do and how to deploy the app directly as a process.
> In your scenario, I spend learning 50 different technologies that I will never use because they are not hired for
I'm actually saying the opposite of this. To learn react you also need to know html, css and javascript, so I am saying if you need to make a static website that is just a single page you may as well just write it in HTML. I am saying you should use less technology.
Similarly to learn Docker you need to know bash so you already have the skills to just skip over docker and deploy your app directly by running bash commands on a linux server.
The "50 different technologies" is how I see the flavour-of-the-month cloud platforms that take something that's relatively simple (copying code to a server and running a command to start it) and turning it into a proprietary cloud platform you now have to learn. Whereas you could have just learnt how to deploy software to a linux server instead and have a skill that was valuable 20 years ago, is still valuable, and will still be the way software is fundamentally run in 20 years time as well I am sure. Meanwhile I have no idea what Dokku and Kamal even are - I'm sure they won't last as long as linux and bash have.
I'm sure bash will outlast me. I do hope to work on Dokku for another 10/20/40 years, but if there's a better project out there, I'd be happy to help users migrate and bow out of the deployment space.
I maintain Dokku for the folks that don't want to build and maintain a bespoke deployment process around docker, and the k3s implementation uses helm at its core, so users can eject when Dokku has run its course. I think there is value in that - maybe not for everyone, but at least for folks busy with building product on a budget.
That said, I'm happy to see folks continue to build new tools and iterate on ideas in this space. Definitely cool to see how everyone improves on stuff I - and others - been working on for the last two decades.
I think the gp is saying the exact opposite of what you gave an example for, and in line with what you are saying i.e. less technology.
one more thing is you tried both kamal and dokku, and then made a deliberate choice of sticking to k3s because suits you the best rather than making a mindless descision about using k3s, which most people do!
> One of my coworkers made a completely static portfolio website with a single page and used react.
People love hating on react, but I mean this is pretty common. Raw is fine, but even in a "static" thing, you'll eventually go "ok now I want to reuse some text here.." Or have some common elements with x things shared. So I'm gonna have to write some hacky imperative JS? Or messy CSS? And what starts as static very easily becomes slightly dynamic. I hate overengineering, so if React were hard to setup or deploy I'd agree (eg Kube), but the cost-benefit of React makes it worth it in most cases for me
Yeah pretty much this. I tend to use Astro but I think your point still stands.
I wanted to make a basic online agenda in a few different formats for a Toastmasters club and though I could have used raw html and css to build it, but each agenda had slightly different layout that would have been a lot of copy pasting in html and lots of room for error.
With astro and react islands it was fairly simple to implement.
I think music is a good "test" for logically thinking people because it requires you to put away your logical brain and just accept that music sounds good because "it just does" and music theory is the way it is because "it just is".
I don't think you can get good at an instrument or enjoy music if you keep trying to force it to make logical sense in your brain all the time. The fact that the author admits straight off the bat that he's never managed to learn an instrument kind of attests to this in my opinion.
Fair enough. Models tend to be imperfect, and music theory is definitely no exception to that rule, and we attempt to model music because we care about the underlying phenomenon (music).
As someone who enjoys listening to music, creating music, and music theory, I'd characterize the latter as providing a specialized vocabulary; an approach to listening, since words in the vocabulary represent concepts that you can hear; and a toolbox of concepts you can draw from.
It's a model, you can use it as much or as little as you like. And there's more than one model, and sometimes the different models overlap more or less, and you can choose any of them or none of them and still make and enjoy music.
The other thing it provides is a wildly complex logical system with fascinating philosophical connections. Some people enjoy exploring that more than making music with it, and that's also okay. I've certainly gotten a lot of enjoyment out of that. It's made me a better thinker. In math, formulas can be elegant. In music theory, different constructions induce different experiences.
Fair points, I should probably disclose that my comment is perhaps one of those "looking like I'm giving advice to other people but really just giving advice to myself" kind of comments :)
I have began learning instruments before and fell into the trap of asking too many "why" questions and then just got lost in music theory that a) didn't make too much logical sense and b) didn't help me actually get better at an instrument in any way.
I think it's very unlike programming where you could start with a hello world program and keep asking "why does it work like that" questions and keep getting deeper and deeper and actually end up with a very good understanding of computers by the end of it that actually would make you a better programmer.
But when you're learning an instrument I think perhaps at the start it's better to just accept that "chords sound good because they just do" rather than going off down a complete rabbit hole.
Yes, I agree, the rabbit hole is real. It has taken many people away from their initial intention of making music. If you're a musician first and foremost, music theory can be a trap. Although, Alice in Wonderland also shows us the intrinsic value of a good rabbit hole.
Edit: Also, what you mention about "why" is true. The models attempt to organize observations of conscious experiences into mathematical and logical frameworks, though they cannot ultimately explain the causes for those connections. The causes of the qualia remain mysterious. Musicians tend to be particularly in tune with how their own experiences connect to music, so some are enthralled to explore that connection itself.
I think the internet makes this worse because I've noticed a lot of music content on youtube is theory related. But I think that's probably because of this bias the internet in general has of "you only see videos about things it's possible to make content about". What I mean by that is it's not possible to make a 10 minute video on "just sit down and play your instrument". But you can make 50 videos on theory.
Don't know if this phenomenon has a name, but I've thought about it before. Social media is filled with posts of people buying things. Because it's not really possible to make a video showing off something you decided not to buy. Same thing with music and people constantly buying gear. "I'm using the same guitar I've had for 10 years and decided to just sit down and practice for 2 hours this evening like the 2000 other evenings before" doesn't make good content.
Something along these lines, people who would like to improve their musical abilities sometimes procrastinate by doing music theory instead. It's safe and it's introspective. And it's authoritative---a legitimate and difficult subject---so there's the potential to feel falsely reassured that you're not wasting time. In a sense, it avoids the more difficult act of creation. This seems to be especially more likely for those who would have wanted to write music.
I.e., "drink responsibly." Perhaps music theory classes should come with a warning label, like the Mirror of Erised. You can stare into it endlessly and produce nothing.
with good strumming and fretting technique, you can play any of the 10 most used chord progressions in a nylon guitar and impress A LOT of people
most popular songs on charts today aren't on the edge of what's new... music theory is a rule set and to break the rules you need to know them. there's a big difference in an illiterate and a jazz musician improvising something for 10 min., as well a music from the scratch by someone who understand orchestration and your unknown local rock artist
Being decent at music has definitely made me a worse programmer because I just do many things - software design, structure, names, you name it - on the vibes instead of sticking to the rules to the chagrin of others :D
Sibling comment got the definition, but “TFA” comes from “RTFA”: “read the fucking article”, in response to “yeah, but what about ($THING that was discussed at length in the first paragraph)?”
Origins: “RTFM”. “What’s the switch for ls that prints time stamps?”
Searching for a niche topic on an internet without advertising: 10 results of pages run by hobbyists, who care so much about their topic of interest that they are willing to pay out of their own pocket to make information available to others
Searching for a niche topic on an internet with advertising: 1000000 results of SEO spam, senseless AI generated articles, 10 minute ad-filled youtube videos, posted by people who have no intent to actually provide high quality information but just see internet users as hoards of mindless meat to show advertisements to
It sucks that the middle ground has basically been abandoned. Simple sponsor slots on niche websites that are focused on specific topics. You have a blog about bicycles with an audience? Seems the perfect space for a bicycle related company to advertise on. Work out a deal for a rate and there you go. The brand can figure out if it’s worth it based on referrals or other metrics.
I get it, it involves a lot of manual labor and it messy and it’s easier to just slap some js code and forget about it.
Searching is the key word there. Without ads, Google would be behind a paywall. Without ads, Instagram, YouTube, Facebook, etc. would also be behind paywalls.
The term "unconscious bias" is based on the (false) assumption that every group of human is exactly identical.
It is 100% a political term that is shoved at people to make them feel like a bad person for noticing differences where there actually are differences.
If you want to eliminate "unconscious bias" you are trying to create a society where women talk to men in the exact same way that they talk to other women, women act the exact same way around attractive men as they do around ugly men, adults talk to a 5 year old the same way that they would talk to a 95 year old.
i.e. a delusional utopia that is completely detached from nature and reality, and which has never existed in any society.
"Unconscious bias" is so embedded in corporate-speak today that nobody has stopped to actually challenge its existence or the proposed solutions. Plus the "evidence" for such bias is so shaky ("Bill said I'm raising my voice, but he only said that because I'm a woman") it can be "found" or "dismissed" very easily, depending on what political goals you have.
I find it odd that we attempt to somehow reshape the innate human psyche (which is constantly taking in information and making assumptions based on them), rather than just enforce workplace rules around how you treat people. Seems like a bit more direct, objective and actionable than diving into some deep conversation about "unconscious bias".
But regardless, the real answer why this is being discussed is: 1) HR is told it needs to be done [usually because some other company is doing it], 2) it provides cover again lawsuits if employees sue over workplace harassment and 3) it's a moneymaking business for the consultants that teach these class [a lot of money].
It’s also profitable for both companies and governments (income tax) to make women delay staying with a baby at home and focus on work instead in their most fertile and productive age (late 20s, early 30s). For men you don’t need extra motivation, as they are not making this choice anyways.
They can feed the lie of ,,having lots of time to have a baby later’’, and I’m seeing many women at age 40 realizing that no man wants to start making a baby with her at that age, because it’s just biologically not practical.
Acknowledging substantial differences in people is typically just left as an exercise for the reader, because everyone understands that part from their formative years, implicitly. The part people don't implicitly understand is how their powerful but imperfect approximation-machine brains lead them to treat others unfairly based on differences that are immaterial to a given context. That's why it's taught about.
I think maybe the term "unconscious bias" is misaligned with its use, because the fact is we have lots of biases we don't actively think about, and most of those are useful, not harmful. It's specifically the unconscious biases that are unfair to others that we need to beware.
Unconscious bias is innate to the way we think, it's real, and it's sort of measurable in a way. There's a test called an "implicit association test" where you categorize words as quickly as possible into one or the other group of categories. It attempts to measure the relatedness of categories within a group through the response time of the testee as they sort words. It's something a junior programmer can code up themselves in a day, and having taken the test in good faith and seen the results, I believe it does peak under the hood of how we think to some extent, revealing unconscious biases.
Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
reply