I’m conflicted. I guess we’ll see what the final total is once an enormous level of unpaid human effort is expended verifying the AI outputs. A little sad if that’s the future of math.
It kind of reminds me of when tech giants open source a project as a means of putting a positive spin on abandonware. “Here’s the source! Any problems are yours to fix now. You’re welcome”
Where are you getting unpaid from? Almost all people who are qualified to analyze the results are paid researchers. And if it is unpaid, then it sounds like they're looking it over for their own reasons, and that's fine?
I also fail to see the issue you have with releasing abandoned source. In what world is that bad? That obviously is a gift and should be encouraged. e.g. id software's history of doing that has meant their work stays alive forever.
You have the frontier labs who are marketing that it’s over and they’re building intelligent machines and you’re saying the mathematicians should ignore it? Ok
I'm not sure how you construed that, so let me be more precise.
Mathematicians, as autonomous entities with no formal connection to any AI lab, have zero obligation to do any work for those labs. OAI can't do anything if all the mathematicians band together and say "Sorry, we're not interested".
If they do chose to engage, they are doing so entirely voluntarily, and it would strongly indicate, if they are voluntarily doing it for free, that there is value (i.e. compensation, payment, barter, worthwhile, whatever) to be had by digging in.
Researchers normally don't pay each other to read each other's work. It's a symbiotic relationship. If they think the AI results are nonsense, they could ignore it like any other crank. If they think it seems plausible and it's relevant to them, they can try to understand it. Seems fine?
Why's that? Do you think that institutions and funding agencies won't cover researchers' use of advanced models, or what? The group I worked in in undergrad had millions of dollars of equipment for doing experiments. I'd have to imagine they could get funding for a few thousand dollars in tokens for the theoreticians to have AI assistance.
Okay? Again, when I was in my group, money would flow to Nvidia to buy GPUs that we would use to run simulations (in addition to the experimental devices I mentioned). The researchers could also request money to use for openrouter or build their own GPU cluster if everyone thinks it's good use of funds.
Shouldn't we pay for the best tools if it helps researchers to be more effective?
The symbiosis here isn't I'll read your paper and you read mine. It's I'll publish my results and you publish yours. Groups share results and cross-pollinate ideas. If people don't want to read OpenAI's papers, no one's going to make them. If no one's interested in the ideas that OpenAI is coming up with, everyone can just ignore them.
The problem people are having is they clearly are interested. They think the ideas are good. In fact, too good. If they thought otherwise, they would just say it's all slop, no one cares, business as usual. And you can tell that there's this phase change in their behavior because previously you could ask chatgpt about math and it would just give you word salad and everyone knew that. Now we can all see that it's not just word salad and people are scrambling to figure out what to make of that. Obviously an accurate answer oracle is still strictly useful even if it makes no attempt to tell you why (you can even use it only to help prove your boring technical lemmas when you have ideas!), so obviously this is an emotional reaction, not a rational one.
yes, but in this case, openai does a lot of PR how their models solve important math problems. if it goes unchallenged, parts of society would think that is true. what would happen if we scale it and 10k companies dump 10k papers every month claiming solved math problems. how is this scalable?
We need the companies to humanly review their papers. in the same way as at other companies we use humans to review the papers.
Well, per another comment in the thread, some 20% of their solutions come with formalization, so there's a very high chance (probably higher than typical asks of research mathematics) that they did solve the problem. And that also presents a pretty easy solution to the scaling issue: demand formal proofs.
(If you're going to object that it's difficult to validate the statement of the problem, please first state your level of experience doing so. It's getting tiring seeing people raise this objection and claim that a statement is just as hard as a proof over and over who don't seem to actually know any math and have never tried to write anything in Lean)
> Almost all people who are qualified to analyze the results are paid researchers.
This _might_ have been true somewhat in the past (although it wasn't), but it's completely false today. Anyone with access to a sufficiently advanced model has the capabilities of analyzing these papers/proofs. It's no different than reading a codebase you might not be fully familiar with, and checking it for correctness (give an engineering analogy).
This hardcore gatekeeping of math (and by extension STEM) fields MUST stop.
> Anyone with access to a sufficiently advanced model has the capabilities of analyzing these papers/proofs.
But they don’t. What they have is the ability to ask something else to do the analysis. It’s an important distinction. If the asker has the skills to evaluate the results, that’s one thing, but too many don’t and act as if whatever they got is unambiguous truth.
> This hardcore gatekeeping of math (and by extension STEM) fields MUST stop.
What must stop is the overuse of the word “gatekeeping”. Anyone is free to study these fields and work on problems. What people rightfully object to is uninformed research flooding everything with hard to verify junk.
The discourse around "gatekeeping" is so poisonous. I think a lot of non-mathematicians are looking at this situation as if this was a guild of medical doctors or some other profession with a regulatory monopoly trying to "gatekeep" access to math from a new upstart in order to preserve their own pricing power. That's not how the math community works, it welcomes all and there is no barrier to contribution.
Mathematicians do have some vested interest in keeping the profession from collapsing into an intellectual oligopoly, where one or two commercial players with early access to their own internal models continuously scoops everyone else and pollutes the field with externalities, and the profession itself collapses, only leaving AIs and hobbyists able to stand. At that point it'll be the AI firms who become rent-seekers. That's not really "gatekeeping" in any conventional sense of the term, it's protecting a healthy economy of ideas and the long-term development of mathematics.
I mean, I have a decent math background, but I would struggle greatly to attempt to even tell you what most (or any) of the conjectures in e.g. number theory are about at even the highest level. I can't imagine a layman would have any hope.
Like I was reading some about adele rings last night, which is already going to be quite a concept for a layman to be able to even slightly describe. Then you can layer on that apparently they're locally compact, so we can talk about harmonic analysis on the additive group. Like, come on now, 99.99% of people have no hope of ever following along, and this is stuff from 75 years ago.
I’m unsure. The whole academia is built on unpaid human efforts. Journal writers are unpaid, and institutions paid for their papers to be published by for-profit publishers. Journal reviewers are paid the bare minimum, certainly unproportional to their efforts and expertise.
The product at the end is important, but so is the process. Few of the things that would happen along the way are happening here, so it's harder to justify the value of the deliverable when there is a failure.
Exactly, the scientific method is what it is for a reason. Sure the institution of academia around it is not perfect, but in general, science, especially general research such as this is not about just bragging about how many papers you have published, it is a process that might help us find out things we might have not known otherwise.
Exactly, I was glad to see these withdrawals, its a natural part of a healthy ecosystem of scientific review, hypothesis, claim, test, refute, extend, withdraw, its the heart of science.
IMO If you take out all the stupid human aspects mostly related to fear, egos, etc, we should brace the imperfect and helpful tools, whatever they are, improve them so they are as easy as possible to review, and keep that core scientific discovery loop going
Exactly what Im talking about! This is a perfect example.
In this case we are talking about using AI to accelerate human progress in mathematics and scientific discovery, and rather than stay on topic, a human comes in with a wealth inequality complaint.
The distributions of the gains of AI advancement is a separate issue, we definitely shouldn't hold up progress on the frontier of human knowledge because of wealth inequality. Its an important issue that needs to be solved, but pausing or slowing progress at the edge of human discovery because of wealth inequality concerns is utterly insane.
For people like you there's no such X in "we definitely shouldn't hold up progress on the frontier of human knowledge because of X" that would stop you, at that is what the actual utter insanity is.
Extrapolating the rate, there won't be any left by the end of next quarter. I jest, but reading these is arduous and finding holes in them is going to take time for anyone daring to.
3 non-formalized papers withdrawn, 6 non-formalized papers formalized. Extrapolating the rate there won't be any non-formalized papers left by the end of November, but because 2/3rds of them will remain as formalized papers.
well, it might be that these proofs are correct or it might be that people aren't bothering to spend a lot of time checking whether they are correct. OpenAI already has a pretty bad reputation in the mathematics community for how they are approaching this process, they seem to be more interested in creating a story for their IPO than advancing math.
I can assure you that a lot of mathematicians are spending time on this, but research level maths just don't move that quickly. As of writing this, the initial drop happened 41 hours ago, that is under normal circumstances not a lot time for reading and understanding a proof paper, and it is most certainly not enough time for publishing a rebuttal. One does not claim that someone else's paper is wrong lightly, you sleep on it, you discuss it with colleagues, you discuss it with the author (not sure how that part works in this case) before posting stuff on the internet that you might come to regret.
But can the others even be "disproven", given that they apparently are so messy and awful that no humans can follow them? Shouldn't the onus instead be on OpenAI to prove that they're right, instead of hundreds of mathematicians wading through slop?
Retraction and withdrawal are different. Retraction is when you publish something, it passes peer review, is published, and some time later its publication is undone, often by an editor or some other person because some fraud was uncovered.
Withdrawal is akin to submitting a paper to peer review and then when you’ve noticed mistakes, you decide to take the paper back and correct it.
Reject is when someone else notices the mistakes and tells you to take it back and correct it.
Withdrawal and reject happen all the time in a scientist’s career. They don’t necessarily mean the scientist is doing bad research, just the research was not ready. Retract usually means something more.
By dumping the papers, OpenAI skipped the typical peer review process, so peer review should be understood as what’s going on now as mathematicians look over the papers and find flaws.
Except Mathematics and science is not done this way, its not code that you can just release bugfixes to, its not a numbers game, its about furthering our shared knowladge. If it is done by flooding everything with a bunch of paper that have not been peer reviewed and verified, and are known to be error prone, that just takes away a bunch of mental capacity from scientists, and time, to manually verify all 400 of them. The issue here is how openAI approaches the science, not their hitrate.
I got the impression that indeed mathematics is done exactly this way. People used to write proofs, somebody would find issues in them that don't invalidate the entire work (or they do), the mathematician would work to correct the issues and resubmit.
