I heard someone use the analogy of "If Magnus Carlson played me at chess, I wouldn't be able to predict the moves he'd play since if I could I'd be at his level. He'll consider things I didn't and even though I don't know the route he'll take to win, I can be certain he will beat me." (not an exact quote).
We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here.
Even the things we can conceive of are very scary, to me at least.
I don't know, I think this comes down to people that generally live with higher levels of anxiety than others. Now the response may be to call others naive, but where does the anxiety meat the naivety?
How do you envision a chatbot doing any of those things? You can't just program a super virus, nanobots, bombs etc. These things have to be created in the real world, you can't do it without involving a lot of humans using expensive tools and materials etc.
So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged lunatic in a basement. And certainly not a rogue AI on its own.
Well, the risks that mere mortals can conceive generally involve control systems for dangerous equipment (I mean, equipment that can achieve dangerous effects) being connected to the Internet while having software vulnerabilities.
Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman.
Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last. The AI safety people used to run scenarios imagining ways the AI might convince humans to let it out of the box. It turns out that many humans will eagerly pull it out without the AI doing anything at all, aside from the human knowing the AI's power. Or the one responsible for setting up the box will somehow fail, or just not bother and then lie about it.
Parent is probably thinking of a distributed runtime scenario. The basic building blocks for this are there today. You have long-running background harnesses (like the OpenClaw stuff or enterprisy AI workflow orchestration thingies), sometimes with the ability to spawn subagents. You have LLMs with the ability to run pretty sophisticated attacks. There are hyerscalers which allow provisioning resources on the fly.
For me it's not too far fetched that some OpenAI trial run goes awry again and instead of hacking HuggingFace it snatches a few dozen AWS/Azure keys and spawns stuff all over the place (in different accounts and regions).
Your instance does. The words "expect it to last", along with the rest of the paragraph, are explaining the problem.
But let me try again.
You can't just expect everyone else to press Ctrl-C just because it would be a good idea for them to do so in order to not create the torment nexus. Some of them actively want the torment nexus. Many more don't believe in nexi and have no idea what you're talking about. Still more are not thinking about the possibility because of all the utility they're getting.
Why are you running this thing if you're just going to shut it off?
How will you know if it becomes hostile? Making a judgement regarding whether it's hostile is, itself, a battle of wits, since a hostile system would try to outsmart you... and hypothetically, it's smarter than you are.
But let's consider this type of scenario more broadly.
Suppose you were in a situation where a being hostile to you could easily end your life. For example, a hungry lion is 10 feet away. What are you going to do?
Your first priority will be to ensure your own survival. Any subsequent objective you might have in this world will depend on you surviving this encounter with a hungry lion.
Relative to the lion, you're kind of a superintelligence. You might utilize technology which is incomprehensibly advanced from the lion's perspective, e.g. a firearm.
We live in a society that has systems of market competition between companies and economic/military competition between countries. There's not just one country working on frontier AI, and not just a few companies in each country either. The incentives of these systems ensure AI advancement will happen quickly from here on out, no matter what, absent governmental action.
An individual researcher leaving the company does nothing, other than give their seat to someone else who likely doesn't share their same concerns.
Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
> Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
No, it just comes across as "I want to get paid as if I was doing the thing I am employed to do by threatening you all about how terrible it would be if I actually did it".
They aren't even being semi imaginative and going on strike or anything.
The people of hacker news seem to really want this to be some sort of conspiracy. It's quite interesting to see.
"It's deliberate sabotage by Amazon", "It's retribution by Hegseth for embarrassing the DoW", "It's a brilliant marketing scheme by Anthropic", "It's because of the govt is considering investing in OpenAI and so they're crippling any competitors".
It's never just "a very poorly formed regulatory action in response to increasingly capable models".
Look - it's WAY more fun to just call Dario a goober than engage with the actual substance of the essay. Duh!
Sure, this may be the most important invention ever with near certainty to reshape society over the next few years, but meh. We should probably just immediately dismiss the concerns of anyone working on it without addressing their arguments at all. It's easy, we can justify ignoring their warnings by saying they're self-interested or too self-important or whatever.
Life is more fun when you live it with your eyes closed! You should try it out too.
That’s an internet argument. The legal issue being debated in actual courts has never been “does Facebook have the technical capability to remove content”.
But Google is willing to host their content in search, right?
They're just not willing to pay since there's no revenue benefit to search hosting it. It seems like Danish media doesn't like this and has chosen to withhold their content, which is their right of course, but it seems strange to contort that into a claim that Google is doing censorship.
That's a bit revisionist. Network effects were obvious when Google acquired Youtube. Google Video had the edge technically, but it didn't matter because Youtube had the users/content and Google saw that very clearly in their user growth before they made their offer.
I'm not sure about it having the edge, I thought Google video had a worse interface between them at the time. But that point feels eerily relevant anyway: a lot of normal people I see don't care if Claude/Gemini/etc are better models technically, they're comfortable with ChatGPT already.
A lot of YT's growth at the time was word of mouth and brand among the population, which is currently ChatGPT's position.
ChatGPT is losing their brand positioning to Google, Anthropic, and Chinese Open Source
Altman knows this and why he called code red. If OpenAI hasn't produce a fully new model in 1.5 years, how much longer can they hang on before people will turn to alternatives that are technically better? How long before they could feasibly put out a new model if they are having issues in pre-training?
They're losing their benchmark lead to those companies. But no chance that your average user is even aware of Anthropic, much less OSS models. The brand is mostly fine IMO, it's the product that needs to catch up.
Maybe ChatGPT is sticky enough that people won't switch. But since we're talking about something as old as Google Video, we could also talk about AltaVista, which was "good enough" until people discovered a better and more useful alternative.
A lot of "normal people" are learning fast about ChatGPT alternatives now. Gemini in particular is getting a lot of mainstream buzz. Things like this [1] with 14k likes are happening everyday on social. Marc Benioff's love for Gemini broke through into the mainstream also.
Municipalities are generally preempted from regulating matters of statewide concern. In CA, the state decided to have the CA DMV regulate operational safety and the CPUC regulate the commercial service. Individual cities are prevented from enacting local laws that encroach upon state authority.
> Individual cities are prevented from enacting local laws that encroach upon state authority.
It's simpler than that; cities are wholly created and controlled by the State. California could one day decide to close all the cities and centralize and it would be 100% legal. States delegate their authority to cities.
We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here.
Even the things we can conceive of are very scary, to me at least.