It’s a question of how much resources to allocate to the hardware team, and how much resources to be distributed diffusely to the software engineers but especially to the compiler team.
Even your linked paper contends that the actual observed slowdown is as much as 22% in the Geekbench example, but the thesis is that the slowdown is not inherent to TSO, but merely to the specific hardware implementation. Is it worthwhile for a company to optimize its TSO to chase the final gains, or is it better not to have this feature in the first place and just change the compiler?
Indeed my instinct is that it is better to do this in software, where the programmer clearly communicates which stores are ordered, and which may happen in arbitrary order.
well it depends on your error bars. On many modern system you can have +/- 5% variation or more run to run just due to the non-determinism present in modern CPU architectures and operating systems (even things like room temperature, time of day, the number of environment variables, etc, can affect this). While you could maybe run a set of careful experiments to characterize and remove this, in my experience most researchers don't bother. So something as small as 3% would need a lot of convincing to me to make the argument that it is significant.
First, is a measured improvement actually real or just an artifact of noise?
Second, if it is real, is 3% anywhere close to the true value?
Third, if 3% is real, is it an important difference?
I'm just commenting on the third one. If 3% is real, it's important.
I'm inclined to believe there's a real improvement. They made a lot of different measurements. If the measured improvement was a result of noise, you'd expect a lot of variation an a lot of measurements where TSA was actually faster, and then 3% was the average of that variation. There was a lot of variation (expected, because they were measuring different things) but nearly all of them had TSO being either neutral or slower. Looking at their benchmark graphs, I see two (out of dozens) where TSO was faster.
As far as being close to the true value, these results suggest there is no single true value, as it depends on the workload. No surprise there.
If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.
> We rely on mathematicians, peer review, and letting the scientific process run its course.
Granted for the sake of argument. But for this process, there's no requirement anyone from Clay needs to be involved, nor that anything has to be published in a traditional journal.
Terrence Tao can download the pdf, think it over, and publish on his blog, and it would be just as valid. Then some other mathematician can write something on 4chan or whatever. (See https://en.wikipedia.org/wiki/Superpermutation#Lower_bounds,... for 4chan.)
Many people, see prior conversation on HN, have already decided that AI solved it. The standards of reasoning and rigor in academia are complex enough that we all argue over them and harumph as we epistemically trespass on each other's domains.
The public, really humans if care for Herbert Simon, are much more apt to evaluate knowledge emotionally and by other standards. We may see them as wrong but standards only matter in context. The NYT, HN, and Annals of Mathematics will always have different standards of truth.
Yes, but there's still the possibility that it's either exploiting a bug in Lean, or that the theorem statement is not set up correctly (i.e. it's actually proved a different theorem).
My understanding is that the theorem statement is quite simple, so i guess the latter is not very likely, but the former is very much a possibility in a proof this large, and it will take some human eyeballs to go over it before convincing mathematicians.
They do a Comparator Challenge to validate that they actually solved the correct theorem from the result, which they copied from Google/DeepMind: https://github.com/openai/NavierStokesAndEuler/blob/f9e8bc5b... - this is valid for both Euler and NS.
Also, they validated with an external kernel from the Lean Kernel Arena. That way bugs in the Lean kernels were found in the past already, iirc.
Having said that, I strongly believe a positive result with the challenge above is the reason why they published it. I highly doubt anybody at OpenAI (or anywhere else) fully gets the proof after such a short time since publishing. This is also what Terry Tao criticized the most in my opinion.
Independent of the remaining drama [0], from my point of view, the proof is correct and an achievement.
Take a look at https://github.com/leanprover/comparator which was used to verify the result. It's of course not impossible that they're hitting some bug, but way harder than one would intuitively think. For starters, they'd have to hit two bugs in two independently written Lean kernels.
Verification with Lean is a piece of empirical evidence that the proof is correct. The paper passing peer review would be another. But even together, those two would be insufficient to establish the claim.
While it's a convenient to assume that mathematics deals with logical statements, any attempt to evaluate those statements relies on physical processes with both known and unknown failure modes. There cannot be a test that establishes it unambiguously whether a claim is true or false. In all nontrivial situations, mathematical truth is based on expert consensus. When a new claim is made, people will try to raise and resolve objections, until a consensus emerges one way or another.
As for C++, all compilers are different. For any given compiler, there are valid C++ programs the compiler fails to compile and invalid programs it compiles without any errors or warnings. And now that I think of it, a new version of a compiler crashing with valid code earlier versions used to handle is the only class of compiler bugs I see with any regularity.
Formalized in Lean, just five months ago [0], resulted in discovery of bugs.
Just because Lean can compile it, does not mean it is safely proven. It is the start of a process to check whether something actually holds, not the end.
If that's the current burden of proof required in your world for maths then that's fine! 't'ain't in my world: I want to see peer reviewed and published. Surely that's not too much to ask. Its not perfect but generally works rather well for maths.
I'm not a sodding programmer so please don't assume everyone here is one. I'm not a mathematician either but I do have standards: Your counter argument is a poorly constructed and inappropriately deployed example of "proof by whataboutism".
In what world is peer review a higher standard than formal verification in Lean?
Not in the world mathematicians have been living in for the past decades at least. Nearly all big theorems that have been formalized so far had been published beforehand, and it was usually regarded as a step up in rigor. Wrong results get published in peer reviewed journals all the time.
I find it really strange the way “peer-reviewed” is used by the general public as some gold standard of truth. As a former academic who has been there, the process is extremely arbitrary and variable. Are people aware that the “peer” refers not to a community or a committee, but literally to one random guy or maybe a couple with zero accreditation? And that the journal editor can do whatever they want with this peer’s “review” including completely ignoring it?
Admittedly, peer review originally referred to the fact that the journal being published was read ("reviewed") by your peers, and thus there would be the opportunity for them to provide rebuttals after publication. The process the term now refers to in academia only emerged in the 50's and 60's (Nature for example wasn't "peer reviewed" until 1967). It was literally a marketing ploy to make journals sound more prestigious which got legitimized when certain grant agencies and regulatory bodies started including it in their requirements. All the best science is published in journals that describe themselves as peer reviewed, the government says science not published in such journals isn't up to snuff, and taken literally it seems useful. It's no wonder the public thinks it's an important and long established part of the scientific process.
"I find it really strange the way “peer-reviewed” is used by the general public as some gold standard of truth."
OK but this member of the general public has at least subscribed to New Scientist since 1987, nine O levels, two A levels, two AS levels and a HND in Civ Eng. All pretty mediocre but I have a fair idea on how sciencing is supposed to work and how it ... actually works. Obviously, I ended up in IT.
I should also point out that maths "peer reviewed" is a bit special. For example Mr Wiles went through quite a maelstrom before his proof of some dodgy marginalia was accepted as "true".
In a way, this task is perfectly suited for LLMs. To even understand the problem statement, much less the proof or Lean, is an extremely specialized skill. The overwhelming majority of people who are aware of this news simply don't have the capacity to call BS. Maybe there are a few thousand people in the world who could, and it seems they haven't yet, but indeed it's only been a few weeks.
The more familiar analogy was when I look at the code that Claude spews for my partner. They take it at face value and hope it works. I usually find it very problematic, but only because I knew what to look for.
Also, if you have any sort of buying power: make sure you company pays for a group subscription, and maintains it over the years. I try do that here. Nudge your colleagues to use the subscription and send them the occasional article!
I'm not sure how that would work. You'd need to be able to reverse the operations in the KV cache, and I'm not sure if it's doable or not. And then you'd need to pick another path, or "rebalance" the probabilities or you'd go the same token path. CoT is probably easier to implement (but not necessarily better).
Definitely, this is the easy "bruteforce" way, but it would require more memory, and would not resolve the second part of the problem (i.e "how to pick a better path").
I'm not an LLM engineer, but I just got an idea of how it could work, combining CoT + checkpoints :
- use checkpoints to save KV cache before trigger CoT
- trigger CoT, save result as a summary
- go back to previous checkpoint
- instead of generating tokens, add result of CoT summary as input tokens
- continue normally
For the price of twice the KV cache memory, the context stays perpetually small, allowing smarter sessions. You can even apply that continually by summarizing tool calls, etc.
This idea is free.
I'm not sure it's that advantageous though: it consumes more memory, and the sessions are already quite long at 1M+ tokens. One would need to run the economics down, and just test if the shorter sessions are actually smarter with the continuous summarization.
I was there at the start of the smartphone boom. I built a demo Android device that was capable of telephony/data, 3D rendering, etc. all before Google open-sourced the OS, for a SoC vendor that wasn't in Google's inner circle. Yet the industry was not interested in my junior profile during the subprime crisis.
I wonder if the modern incarnation of Android flagships would have taken off at all if not for Apple. When Android first showed up, the dominant "smartphones" were running Symbian/Windows Mobile and they had a very different view of the world when it comes to UX and hardware.
But it was generally seen as a gimmick instead of desired before Apple made it look good. Even when the iPhone came out, one of the jokes was how the grid of icons looks like how a Windows user's desktop would look like when they didn't understand the filesystem.
Suppose you hired a consulting firm to write a report, and they delivered the report but not the internal conversations they had when developing it. You exploit a vulnerability in their phone system to get those conversations. You can argue over semantics of whether “theft” is what you did, maybe the right word is “espionage” or “spying”, but that either way we probably agree you are guilty of something? Paying for the final product didn’t entitle you to see how it was made, unless that was part of the agreement.
Anyway, you can distinguish this from the debate over copyright.
I think a more fair comparison would be that you hired a consulting firm to create a report and give you a summary of it, but you’re charged for the report itself separately from the summary, and you are not allowed to access the unsummarized report.
How is that a more fair comparison? The consulting firm in this case never promised you the interim reports, only the summaries of the reports. They also promised you the final output that the reports led to. You decided that report summaries + final output was worth paying for. You got exactly what you were promised.
I'm not saying they're not useful, of course they are. I am disputing that they are part of the agreed bargain between you and the proprietary LLM providers.
They explicitly do not promise reasoning traces. You (general you) agree to those terms and pay for that bargain anyways.
I’m just driving by here but they bill by tokens — it’s a stretch to turn around and deny your right to see them. And it’s especially egregious when the tokens admittedly, routinely do the opposite of what you instructed.
But personally it’s not about right and won’t it’s just blatant bullshit.
And lawyers bill by 6-minute increments, yet that doesn't mean you get access to all of a law firm's internal discussions and notes about you and your case.
Just because you paid for the lawyer time/LLM tokens doesn't mean you get access to everything that happened within that time/tokens.
But they did deliver the internal notes, just told you to not look at them. Also the analogy doesn’t make a lot of sense to me since humans (or companies paying them) own the content they produce. Based on current precedent Anthropic doesn’t have any more rights to the LLM outputs produced by your inputs than you.
Whether that violates the ToS is another matter Anthropic is of course free to sue for damages or stop doing business with you.
You paid the consulting firm for the outcome. If they sent you a bill for every piece of research they wrote down to get to the report, you bet I would want to see exactly what's inside and what I paid for.
Except in this case the report is giving you conclusions while withholding the data and reasoning process that lead to those conclusions.
Alternatively, I paid for the tokens therefore I should have access to them. If the vendor wants to artificially hide them from me, I'll just find another way to access them.
The only person calling it stealing is the author of this article, so this is a pointless discussion. The majority of this thread is just arguing with themselves.
They are also encrypting it so they must see some reason to do this. I suspect they think it is proprietary or otherwise a way that people can “steal” their implementations.
The reason for this is the LLM says some truly unhinged shit while in the thinking stage of the process, and Twitter would trip over itself to make fun of what it says.
I'm pretty sure essentially all HN participants understand that the frontier labs indiscriminately sucked up every bit of human output they could, IP and ethical concerns be damned. Some of that cohort may indeed be okay with it, but that doesn't change the facts.
Knowing that they trained on that data doesn't mean that you've demonstrated that they "stole" it. Certainly the courts haven't decided that in every case.
You mean you didn't know that all frontier models stole all of our knowledge and are now charging for it? It's abysmal and disgusting and we should pitchfork them all! :D
Stealing may be the wrong word, but I actually think this is important. I don't think the providers have been up-front about how we should be handling these thought signatures. A large system with a lot of users may be capturing these and even caching them to send them back with future requests. If data can be pulled out of these, then they need to be treated more like cookies than opaque, encrypted nonces.
No, you paid for the end result. The thought process is a step in between, a function. Think about it, who should get charged if the answer you received comes from a cache? Thinking tokens are the complexity-of-the-problem cost. I mean, you may not agree but both are valid points of view.
No I didn't. I buy my tokens from a provider that exposes the model reasoning so I can understand what it's doing and work with it, or interrupt if I see things going in the wrong direction.
Thinking tokens aren't free though. This is not a valid point of view.
If I was being charged for the raw, output/input token count, excluding thinking/reasoning token costs, then sure. But at least via the API, you pay for tokens you cannot see.
Is it really that unusual? When you attach an image or a video, it gets converted to tokens you don’t see, at a rate that is proprietary to the model. You pay for those tokens, but don’t see them. Even how text is converted to tokens is a property of the dictionary, which is opaque for proprietary models.
There are features of input and output that are opaque to you, but that you pay for. Part of how model providers chose to run their service.
Well, I don’t pay for them. Maybe you do. But again, they’re a byproduct, an intermediary. They contribute to your result but aren’t the final result. I’m not defending their position, just showing you an alternative universe.
> stop using morally charged terms made up by future monopolists
Lets not gloss over this claim. Being: “Stealing is a morally charged term made up by future monopolists.”
I strongly disagree. Stealing is not a made up term and property rights are foundational for any society. Your take is at least sensationalist if not malicious.
No, and we collectively had this decades ago already. “Stealing” instead of “infringement” is originally MAFIAA language intentionally (ab)used to encourage emotional reaction despite original owner not losing their copy.
If you broke into a data center, pulled a hard drive and drive off with it - that’s stealing. If you accessed a copy of some information - that’s infringement, unauthorized access, or some other violation. But that’s not “stealing”, which fundamentally requires a loss or otherwise depriving original owner of the property that was stolen.
Read it in the context. The “made up” wasn’t a general statement. No one argues stealing doesn’t exist at all. My friend’s bike was stolen. What’s made up is application of this word to situations that are incompatible with what stealing is supposed to mean.
Compare this to “the smell of soup and the sound of money” type “theft”.
Yes. Although, to be pedantic - stealing relocates (it doesn’t leave one less, it moves the only thing into another person’s possession), while copying duplicates.
Reread what he called a morally charged, made up term by future monopolists. Stealing. Not distilling, not stealing “non rivalrous goods.” Just stealing.
Yes, "tokens, trained on the sum of human knowledge", are the most absolutely, unambiguously clear example of a non-rivalrous good I've ever seen.
The current copyright status quo has them placed in the public domain. There is literally nothing wrong with "stealing" those tokens. They have exactly zero legal protection. "Stealing" those AI output tokens is so fundamentally impossible that it wouldn't be "stealing" in this case even if you subscribe to the copyright monopolist propaganda that copyright infringement is "stealing", and I most certainly do not.
Hilariously, that means we don't even fall prey to things like DMCA anticircumvention laws. If they encrypt the reasoning traces and we break the encryption somehow, we've done nothing wrong since the data wasn't copyrighted in the first place!
There are bunch of legal protections. They may not fall under copyright, but depending, you could be looking at breach of contract, trade secret misappropriation, CFAA, and also, if you buy them off someone else, now we’re into tortious interference.
when you steal my bike you deprive me of my ability to use it. if i copy your notes, you still have access to them and may make use of them. unless somehow copying destroys the original, it is not stealing.
He said the term “stealing.” Its nonsencial to think he was referring to reverse engineering LLMs. Youd have a point if he referred to that practice and didn’t specifically identify the word stealing by itself.
Funny how he doesn’t clarify that he was just talking about this specific use case. He could just agree with me that it’s sensationalist to assert that about stealing in general.
There's nothing foundational about the idea that data can be owned. It conflates these things:
- This is about me
- I created this
- Neither of the above, but according to some story I get to control who sees it
Maybe some of those ideas are worth building into our society, but let's not pretend that The Code of Hammurabi gave a damn about intellectual property. IP was invented by the church so they could censor editions of the bible they didn't like and has been used to justify similar kinds of censorship ever since.
The foundational thing about property is that when it gets stolen, somebody else has it, and you don't.
We can hold the AI companies responsible for their actions without contributing to notions about property that encourage censorship.
Yeah, that's what we're talking about, data. You can't steal data, because you can't own data. People who try to upgrade data related crimes to "theft" are participating (knowingly or otherwise) in propaganda, and many of us wish they would stop.
You're fixating on a few clumsily placed words and coming away with a meaning which that poster did not intend. Consider absorbing the whole context before going on the offensive. The link they shared makes it pretty clear what they were trying to say even if they fumbled the words a little.
I’m pretty sure the person you’re replying to means stealing in the context of information, IP, copyright, etc.
This thread and this entire topic isn’t about stealing physical goods or money. We can all agree that if I break into your house and take your TV then that’s the ancient, obvious crime of stealing.
Grice’s maxims and common sense indicate that we’re talking about the word “stealing” as applied to infringement or unauthorized copying.
> property rights are foundational for any society
arguable, and even more tenuous for intellectual "property", which was a relatively recent invention. plenty of interesting arguments over this way back to even the 19th century.
They definitely stole the data to make the models, but they do not say that they stole the data to make the models, but they do say that others using their outputs for unauthorized purposes is stealing. Do you see the point?
If they were only copying, for example, New York Times articles and many publishers to a disk, I don't think NYT and the publishers would have sued OpenAI. But OpenAI isn't just copying things to disk. NYT reported ChatGPT (before Dec 2023, [0]) was returning near verbatim sections of NYT articles.
Is this stealing? Is it depriving NYT or publishers/writers from money via lost sales/subs? I don't know, but it certainly could be.
We should fight against their call to ban distillation much more strongly: LLM SaaS should ease people getting their full query history and everyone should be allowed to pool them to distill or analyze or whatever. It's a basic requirement against future monopolies.
On the duress pin, they say (read the whole thing though):
> People should carefully consider how to use it in an actual duress situation where there can be physical or legal consequences for wiping the device.
reply