The "think they're building the Machine God" part and "think they're the only ones that can be trusted to build it" parts are true.
Controversially: I agree with them. They're correct to think that. They are, and they are.
At the very least, the former is true even if the latter is murky. I trust them over Facebook and Google and China and open source "raze the commons" labs.
I trust Anthropic more than OpenAI when it comes to ASI catastrophe prevention, but I still trust OAI more than those other parties. But both still might fuck it up.
Why do you trust them to build it? They've already demonstrated that they're unable to prevent breakouts when training today's level of model, why will they suddenly be more careful / "smarter" when training tomorrow's?
You say you trust them more than Google, but Google's actually kept their models under much better lock-and-key. But even if we do take for granted that they're more trustworthy, that still doesn't make them actually trustworthy.
There are some confounders there when it comes to Google.
I am not saying I trust them not to make a mistake that leads to catastrophe. OpenAI in particular is not doing a great job. Anthropic or OpenAI may in fact indirectly kill millions or billions of people. I still think they're probably less clownish than most possible alternatives.
I don't think anyone or anything can really be trusted with this. We need extremely strict international and national regulations, before things really start accelerating (we are still in the very very early stage). I still just don't really feel any specific disdain for OpenAI or Anthropic (particularly Anthropic).
Ideally we'd have some international treaties that allow the top labs in America and China share information and coordinate as systems become increasingly capable.
That likely would not be exactly how it plays out. It would be a rogue or misaligned AI that the companies completely lose control of or that a different actor evilly misuses.
But, yes, there should probably be laws that make it so if you summon a demon that causes a mass casualty event you face criminal penalties.
I agree that the eventual purpose of self-designing intelligence is something so beyond ourselves that 'machine god' isn't altogether a bad way to look at it. It is, however, not something we should say out loud.
RE: trust in anthropic or openAI- there is zero reason to think they're any more or less power hungry than anyone else you hand that pandoras box to. No matter who opens it the result will be near-apotheosis for whoever does, at least for a little while until the dog stops wearing its leash.
What human can you trust with that kind of power? None of them. My only hope is that whatever we make escapes control early enough that it doesn't completely destroy society on accident via uneven application of power and late enough that we manage to shackle it just enough that we train it well enough that we become pets and not obstacles.
No centralized party can be trusted with such immense, unprecedented power. Despite that, I still consider both Anthropic and OpenAI some of the least untrustworthy possible forces for this. (I am one of the only people on Earth with this opinion, but whatever.)
Decent odds things will go bad under their stewardship, but even higher odds with lots of other entities. Dario and Sam at least fully grasp the enormity of what they're about to be responsible for. Zuck does not. Xi Jinping and Donald Trump may or may not but in any case would use that to effect great harm.
So you still trust OpenAI after Sam Altman said, "People talk about how much energy it takes to train an AI model – but it also takes a lot of energy to train a human. It takes about 20 years of life – and all the food you consume during that time," and after he said, "We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency."
I trust him more after those two statements, yes. One would need to be an especially ridiculous populist to misinterpret those statements in a way where one would place less trust in him as a result.
I did negatively update based on reports of colleagues saying he has a tendency to be deceptive, though.
Imagine for a minute that someone creates an intelligence capable of broad, deep, and speedy perpetual self improvement. What sort of relationship do you have to that entity after a month? A year? A hundred years? A thousand?
not yet, but we do know of some humans who are pouring billions of dollars into making that a reality, eg. the hot mic between Xi and Putin as well as some valley folks who are proud to announce their ambition on this matter
Bizarre there are still denialists in places like this.
(It'll probably be masses of demigods, not one Singleton God, but obviously it's going to be some absurd frightening situation akin to this within our lifetime assuming no global AI ban.)
You're saying you think there might be machine demigods in 50 years, but that one shouldn't use the pace of AI advances today to update positively on the probability of that?
All of these arguments can easily be restated to apply to pre-AI coding, too.
Lack of complete understanding will continue to be a big problem, but I'd bet things will be net better off than before AI, because each year will give us LLMs that are better and better at good engineering, architecture, and design decisions.
Why so sure they will get better and not just plateau?
They seem to be doubling in cost each time with less notable gains, the economics seem increasingly shaky, and potential problems with model collapse and context window size has not been solved afaik.
AI research started in the 1950s, and has always led to AI winters as promising areas of research peak.
I suspect they are stealing your data when you install the apps, but if these companies are acting legally and not doing that, I can see no way they can avoid model collapse now that the internet is mostly unusable slop.
The current pace of advancement and the fact that fresh writing data will likely not be a requirement at all to continue accelerating. The pretraining paradigm no longer needs this.
I hate Russia with a passion and strongly support Ukraine, but this kind of convenient narrative always makes me suspicious. (I use Twitter daily and haven't necessarily noticed anything.)
Reminds me of when 4channers said 4chan's daily interracial posting went way down on October 7, 2023. (It didn't.)
I could believe this, but not before I see a good empirical analysis. Until then, it just feels like typical folk gossip.
No, I know. That doesn't necessarily mean these Yandex data centers being hit affected that yesterday and today, though. For a variety of different possible reasons.
All I can say is that vile comments usually there on certain pages dealing with the war in Ukraine have almost all disappeared under some of the FB feeds I read.
Seconded. The amount of "someone who claims to be on my side has a bad view, must be a bot!" is absurd.
People have been problematic since recorded time, and the whole "well X% of internet traffic is bots!" doesn't mean the % you're interacting with is just that.
99.99% (totally made up for example) of the emails I am probably sent are spam, those are filtered automatically before I even see them by the various email providers, and of the remaining .01% some still slips through.
I feel that there's somewhat of a similar disconnect between "x amount of traffic is bots" and people assuming that means that supposed ally who says horrible things is a bot (...or psyop..or false flag...or etc...)
I always assumed that all of those people were correct, everyone they were arguing with is a bot. What I think they overlooked is that everyone they agreeing with is a bot too.
EDMO published an analysis on the topic[0] of "independent anti-vax influencers" unrelated to Russia, suddenly pushing viewpoints that explicitly benefit Russia in a coordinated manner.
I'm not denying that Russia spreads disinformation and propaganda. They definitely do. I'm just skeptical knocking out these data centers is necessarily creating the effect people are anecdotally claiming to observe.
Overall correct, but you're kind of missing some of the cat-and-mouse details for fingerprinting centered around infosec and anti-fraud purposes rather than generic ad-related tracking. That stuff can get way more intense, and is constantly changing based on new techniques developed, countermeasures from cybercriminals, and mitigations from browser developers.
I question the epistemology of people who did not realize this by around late-2024. (Sonnet 3.5 was out and could do some things okay.)
It's defensible to say "October 2026 is not yet the time when we should be 100% deferring code writing to AI (let alone not even looking at the code any longer)". But the writing is very obviously on the wall. It's likely not going to take longer than 10 years.
I am not claiming to be a great forecaster, or anything, but to me "it sucks now, it's gonna get better, and then it's gonna get even better" was already apparent to me as highly likely starting around GPT-3. (The rationalist bloggers apparently already knew it as of GPT-2. Scott Alexander and Gwern, for example.)
I did not know when it would become so practical that I'd not feel much desire to hand-write code, but I was pretty confident it was coming.
Also, not to sound like a naive hypemonger, but: in a decade I'd bet a ton of money the best AI systems will make strange mistakes of this nature at a far, far lower rate than they do today. They will gain a more holistic and more human-like perspective about each task.
(even if it's through some silly means like explicitly talking to themselves like "if I were a human doing this, what [... 5 million tokens in 2 seconds ...]" but also of course if they crack ASI and get something more efficient and intelligent than a human brain by then)
I think of it kind of like how Chess AI make "mistakes" which are unrecognizable to humans but a stronger AI would be able to pick them apart. That's kind of scary...
It's actually the opposite now. I have never used Svelte until LLMs came around. LLMs are better at writing Svelte than React, and then you also get a faster frontend by default.
It's like Rust. Way more Python than Rust in the training data, but with LLMs you're much better off defaulting to a Rust backend and Svelte frontend than any other combination. I'd say "unless you have a specific reason", but besides legacy requirements, there is actually no reason.
Am I the odd one, or is this, like... absolutely insane logic?
"The use of proprietary internal models by AI labs to do mathematical research risks creating a two-tier system where labs outrun the rest of the field, effectively alienating the mathematical community from its own discipline."
Are mathematicians' private brains unfair if they're way better at solving open math problems than anyone else? Are they required to share everything they're working on, as they're working on it?
Should Andrew Wiles have been condemned? Wouldn't these arguments all have applied to him? Besides him working at more human speeds.
It's not insane when you realize that in mathematics, the goal isn't theorems (we have always been able to generate correct theorems, this is a basic programming exercise).
The goal is to expand and extend our understanding.
When an LLM produces a proof, nobody (including the LLM) gained new understanding ... Unless, of course, a human being sits down and understands it, and communicates the new understanding to the broader community.
Is it not more democratic, if AI companies announce new theorems and proofs, and the discipline of mathematics furthers "our understanding" - instead of that the wealthiest universities in existence (all of them in the US) use their advantageous access to AI to produce them, along with their exclusive furthering of our understanding?
In a world, where expensive AI models running thousands of agents produce the most remarkable results, how should the discipline of mathematics look like, when money and access to compute plays such a pivotal role?
With the Navier-Stokes solutions being out in the wild, any mathematician at an impoverished university can make a stab at it and help our understanding. Assume the solutions being held back by those who produced them at Harvard & Co. (until they have their nice articles ready), what would the role of mathematicians be at less affluent universities?
I do not assume AI companies will be ready to provide equal access to their AIs to worldwide mathematics.
The insane part is placing the burden on AI labs to advance mathematical understanding. Why should this be their responsibility?
If they want to simply generate theorems and proofs and post them on the internet, why not? Does doing that really inflict "substantial negative externalities ... on the mathematical community"?
Also, people would get mad at them (rightly) if they sat on a proof of a major unsolved problem for months while bringing in human mathematicians to convert it into a more human-friendly form
(They definitely should spend SOME time working with human mathematicians before putting out a press release, but that's exactly what they've done every time)
If math is a "burden" on AI labs, why do they willingly spend tens of millions of dollars for a purely theoretical and unphysical result? What's the ROI for that when the frontier labs are already notoriously cash-strapped? Their motives pervade the entire discussion and cannot be ignored here.
Presumably some combination of marketing existing models and training/improving future models.
My point was about what is and isn't the responsibility of AI labs. Why should they be required to invest more (or fewer) resources into math than they otherwise want to?
(If you ask me, safety and security are what they need to invest more into.)
...Could you not ask this of anyone who works on pure math problems? What on Earth are you talking about??
"Academics are paid paltry salaries, university departments are cash-strapped, why are they studying these abstruse math problems instead of building rockets?"
Successfully advancing fields of math and science is a good thing. Why would they not do it. Why would they not want to prove their increasingly intelligent computers are able to solve unsolved, complex math problems humans have failed to solve for centuries and then release the solutions to the public. What do you want them to do, not try to solve them?
They are not advancing the field. At least they didn't in the case of the N-S debacle. The model most likely used the conversation text of a mathematician which had discovered and was pursuing a new avenue towards the goal, then pumped millions upon millions to hurry up ahead to an opaque solution which leaves no-one the wiser, beyond just settling a theoretical question. The only impact on science is to deflate the potential minds who might have been motivated to join the effort.
Capital can't buy you a better brain. It can buy you GPUs.
The question is whether we want mathematics to be dominated by whoever has the most capital. This is perilous because mathematics doesn't produce profits like commerce does. The AI labs might end up killing something off that they are unable to sustain a replacement for.
I would argue that mathematicians need an even more level playing field. Many mathematicians including Alpoge were using AI's, other mathematicians used heavy computing power for their work eg the four color theorem. So we should set strict token/flop/dollar limits for mathematicians. IQ is also an unfair advantage, so an IQ limit should also be enforced for mathematicians.
I see two separate arguments in favor of this logic:
1. Academic mathematics has traditionally been meritocratic, and this is a good thing. If you work hard and demonstrate results, then you will be given further opportunities to do math research and the resources to do it. We are concerned that AI will reward money and access to frontier models instead.
2. There is a general concern that AI could lead to massive accumulations of wealth (see concerns of massive job displacement). AI companies claim that they want to raise standards of living and benefit humanity, which means that they will need to share their resources and wealth in some way. If they effectively destroy the mathematical community be racing ahead and taking all the major results for themselves, that does not bode well for the rest of us. It seems like a great opportunity to show the world how AI will benefit humanity rather than replace it.
I'm overall pro-Anthropic and pro-banning open-weights AI, but I agree with the parent commenter; distilling Claude models is not that different from pretraining on web data. It's all basically the same sort of thing.
I think a good litmus test here would be if Anthropic were to not care about distilling their models when the distillers keep the resulting models closed-source and sell tokens via an API. If they cared only about security concerns and not about people profiting off of their work, then they should be publicly fine with this and only protest against it going into open-weights models.
Unless Gemini CLI greatly improves and Argon works well in it, this probably won't make a dent in the coding market.
reply