I think we all need to update our priors on what LLMs can do today. Here is one anecdote:
Scott Aaronson says "Dana tells me she now mostly understands the proof of the UGC, and is amazed by the new ideas in it, and wants to give talks about it soon."
Dana Moshkovitz is a professor at UT Austin who is an expert in the area and has been working on solving this exact problem for decades!
Meanwhile you can use the model to help you out as Scott comments "Just now, however, Dana tells me that she’s been asking Astra all day to explain the new proof of the UGC to her and it’s been doing an amazing job and she’s starting to understand the construction."
No one knows the answer. However, trusting your curiosity and going where it takes you may be a reasonable strategy. If the model asymptotes, you will be ready to figure out how to build upon it and if not, at least you will have had fun satiating your curiousity!
Terence argues that explanation of results ("understanding") will be the new bottleneck in math research but I am not sure this is the real bottleneck for progress.
Understanding was critical for the field to progress when only humans were involved but if humans are not needed to make progress, I wonder if we split into two worlds: an AI math-world where amazing new results continue at a rapid pace bottlenecked only by compute/cost and a human math-world where we understand a subset of the AI math-world as a hobby (similar to Stockfish vs human chess).
In some sense "understanding" (understanding if it is true, if it is important, how to use it) is about the only bottleneck in math. Any theorem that you can write down or imagine is already true, false, not provable already. In some ways we can already start iterating through all the theorems. We will never get to the end (or really get very far down the line) and most all of them be trivial (I think the Busy Beaver[1] project is a fascinating example, ymmv).
I am wary of AI in all aspects I am seeing it in but in many ways in mathematics seems to me the least troubling. It will change things in and the field will not be the same. Blacksmithing has not really gone away. You can still work as a farrier, if you like that sort of things. The tools that replaced a man working over a forge with a big hammer are part of a giant industry that is still producing works for the modern world.
Chasing these 'trivialities' is a good thing, imo.
The Busy Beaver game has lead to a better understanding of complexity theory and automata. Also, direct "hands on" work on improving proof assistants and related tools.
Btw, for those who are curious, the Busy Beaver Challenge wiki is a treasure trove of rabbit holes and curiosities:
Wow, this comment thread clearly shows that at least Anthropic has not been a great communicator.
If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk.
One may or may not agree with item 1 but item 2 above should have broad support given the unknown unknowns in play?
This argument is old and goes back to cyber security. Things like government backdoors are the same. It's true centralizing power in the hands of a "good" would theoretically be good. But the key part is the "good". I think after repeated experiences with governments and corporations, no rational person would ever assume them to be good. There are simply too many incentives to be bad.
"We can't trust the government to control nukes! Every citizen should be able to build a nuclear weapon in their backyard with zero regulatory authorization!"
Closed-weight models can also pose biological risk, arguably more so than open-weight models, given the fact that they're being used by state actors (the United States military) to kill people right now.
Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
Open-weight models collectivize access and ability to do more for a greater good, and the expense of a frankly low-risk possibility that some randos want to use it for very bad things.
Closed-weight models keep the control in the hands of the few that actually are doing the harm to the world, and the rest of us have no way to stop it or defend.
I actually trust the "small groups of elites, billionaires, and state actors who have the monopoly on violence" over "all of collective humanity". The people with power want to hold on to power - and they've been doing it very well. Generally speaking, it's in their best interest for the world to not go to total shit, and they work hard to have it that way whilst holding on to what they have. It's the disenfranchised that are far more likely to want to have mutually assured destruction.
I think the general public should not be conflated with 'collective humanity' - it's more like... many pockets of people, some well organized, some not so much. They're not all necessarily working together; we all have our own agendas.
Also, it's very much the asymmetry of offensive vs defensive capabilities that's extremely concerning. A disgruntled, mentally unstable backyard biologist spewing out potentially world-shaking viruses is not something I'm comfortable with. At least when the elites do it, they'll have the antidote ready to sell us (fingers crossed).
Is this a bot? As soon as I linked to this thread in another comment, you show up bootlicking the establishment. Forgive me if you're an actual user, but your account has basically no prior activity and you're commenting on an esoteric thread for your second comment on HN.
In the long term, there is no asymmetry of offensive/defensive capability, see the other reply for details.
The comment I made above the reply you're responding to was at 9 votes then a few seconds after that reply was made, it dropped to 4. Suspicious indeed, I've never seen that type of swing on HN.
> Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.
> And the rest of us have no way to stop it or defend.
If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.
I often see this parallel being drawn between general intelligence and nuclear weapons, and you seem genuinely concerned, so I'll try to explain why this concern is misplaced.
Intelligence is not a single-use weapon of mass destruction in the same way that nuclear weapons are. If I blow up a nuke and it vaporizes downtown, there's no "reverse-nuke" I can blow up afterwards to undo the explosion of the first nuke, or a "nuke nullification field" that I can set up beforehand to protect an area from the effects of a later explosion. But the equivalent of all of these things exist for general intelligence and the adversarial/cooperative dynamics that arise thereof such as bio/cyber attack/defense. We're just currently too stupid to realize how that would work, kinda like how monkeys mostly don't understand how humans resolve conflict, and that's why we see a temporary asymmetry in these strategies.
But it must be symmetric in the long run, because two things are simultaneously true: more intelligent species are able to better cooperate and prosper, and the intelligence difference is very small between individuals within the same species. This pattern is generally true from humans to animals to insects, all the way down to bacteria. There's no reason to believe this deep evolutionary pattern will suddenly become false. It has persisted through plenty of qualitative transitions in the past.
So the primary risk of the upcoming intelligence explosion is not adversarial dynamics such as bio/cyber risk. The primary risk is humanity splitting into two or perhaps more species, with the top superintelligent species preying on the rest (though it's also possible that we become cells in larger government/corporation-sized organisms). Broad diffusion of general intelligence is what can prevent this.
It should not be an easy choice to trust the latter. Sure, "literal terrorists" exist, but guess what? State terrorism (and state-sponsored terrorism) in the form of massacres, genocide, torture, forced disappearances, etc. is far more deadly and far more prevalent than fringe ideological groups randomly attacking civilians.
These states and elites with the access to the scary weapons ARE the ones doing the damage. The call is coming from inside the house.
I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians. By one of the few states who had the power. This is the definitional paradigm of state terrorism. And like, now look at the status quo of nuclear treaties and agreements and proliferation. Not exactly a success story.
It's obviously not worth personally formulating a "defense plan" for an AI-enabled bio attack were I personally to have access to SOA weights, but if history is any indication, I feel pretty confident that the likelihood of that happening remains far greater in the closed-weight, elite-state-access-only scenario than the open, democratized, and collectivized one.
> I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians.
Or, another way to put it: an extremely powerful military technology the careless usage of which could destroy the human civilization was only used in one war in history and killed less than a million people, and then it was never used again over the next 80 years. I think this is a success story for humanity, and the NPT was an amazing feat of coordination.
> And like, now look at the status quo of nuclear treaties and agreements and proliferation.
I mean, sure, but giving every state (much less every person) the ability to make nuclear weaponry wouldn't make things any better.
I think the reason we think of this differently is because you believe that states are by default... corrupt, maybe, or unreliable, or evil, while individuals are mostly fine. Whereas I think that most people can't be trusted to make correct decisions on topics like "should we use this dangerous technology", while states at least have some decent track record at this.
Indeed, the core of this misalignment is that we have a foundational difference of opinion on that.
The only caveat I'd push is not that I believe that "individuals are mostly fine," but that as a whole, separate from the state apparatus, I believe humanity collectively has its own best interests in mind and is capable of achieving more good than states or coercive hierarchies can or want to achieve.
My favorite story when I hear someone dismiss an argument saying 'that is obvious':
Lazarsfeld was writing about “The American Soldier”, a recently published study of over 600,000 servicemen, conducted by the research branch of the war department during and immediately after the second world war. To make his point, Lazarsfeld listed six findings that he claimed were representative of the report. Take number two: '“Men from rural backgrounds were usually in better spirits during their Army life than soldiers from city backgrounds.”
“Aha,” says Lazarsfeld’s imagined reader, “that makes perfect sense. Rural men in the 1940s were accustomed to harsher living standards and more physical labour than city men, so naturally they had an easier time adjusting. Why did we need such a vast and expensive study to tell me what I already knew?” Why indeed.
But Lazarsfeld then reveals the truth: all six of the “findings” were in fact the exact opposite of what the study found. It was city men, not rural men, who were happier during their army life. Of course, had the reader been told the real answers in the first place, they could just as easily have reconciled them with other things they already thought they knew: “City men are more used to working in crowded conditions and in corporations, with chains of command, strict standards of clothing, etiquette, and so on. That’s obvious!” But this is exactly the point Lazarsfeld was making. When every answer and its opposite appears equally obvious then, as he put it, “something is wrong with the entire argument of ‘obviousness'”
Oh yes, I hate this kind of thing so much. People have so many pseudoscientific explanations and theories to support whatever random products and practices they like.
I call these “huh, that makes sense” explanations, since that’s often what people say after hearing them. You even used the magic three words above.
The related phenomenon in biology is called the “just so” fallacy; e.g. “X species evolved Y just so that they could do Z”. It’s a comforting story, but it will often lead you astray.
Edward says that Nate's prediction should be interpreted as:
“If nothing else changes between now and the election, Joe Biden has a 85% chance of winning.” (Silver’s argument)
But the problem is this prediction is not testable as election only occurs once at the end. Thus, one should use Nate's early predictions as pure entertainment.
Nate's final predictions show that they are well-calibrated across all of his political predictions but it is hard to estimate how accurate his model predictions are just for presidential elections given he has predicted only a few so far (unless one assumes all elections have similar uncertainty, which is clearly not true).
Greenwald was not claiming he is breaking the story. He was trying to shed light into the media blacklisting of the story. It is indeed Ironic that his story got blacklisted as well!
I think some editors are perhaps trying to avoid a repeat of the 'Hillary email/Comey announcement of 2016' scenario but didn't anticipate the Streisand effect.
In hindsight, best approach may have been to cover the story, get a sound bite from Biden denying that he benefitted, say there is no evidence he benefitted and leave it at that.
I don't get this obsession with comparing Sweden, a remote, sparsely-populated country, to more dense, well-connected, large population centers. If you really want to compare Sweden, please restrict your comparisons to its closely similar, equally remote and sparsely populated neighbors, namely, Norway and Finland. The 10X difference between them will be immediately obvious.
Finland and Norway have half the population of Sweden.
Sweden has about the same population as Belgium, Portugal and Greece...or Georgia (and Ohio, and Michigan, and...) in the US, none of which are places that people are shy about extrapolating from, when it suits their narrative.
You may not like the conclusions that can be drawn from Sweden, but you can't dismiss it based on population. Try again.
Scott Aaronson says "Dana tells me she now mostly understands the proof of the UGC, and is amazed by the new ideas in it, and wants to give talks about it soon."
Dana Moshkovitz is a professor at UT Austin who is an expert in the area and has been working on solving this exact problem for decades!