Regardless of whether they achieved parity via distillation, or whether they got here via independently constructing a model from scratch, it was always going to end this way for the frontier American labs. Distillation “attacks” are not attacks. The frontier labs “distilled” all existing human written knowledge into their models, there was always going to be a second class lab that would distill that model into a cheaper version of it. There was never any plausible explanation for why this wouldn’t happen. There was never any practical mechanism to prevent someone from saving a conversation and using it to train their own model.
Even if it didn’t happen here, it was still the case that it was going to happen going forward. It was always going to end like this. Invest in the hardware companies, not the model companies.
I strongly agree with the premise that distillation is not an “attack”.
But that said: K3 is not a distilled version of Fable or Sol. Fable has been barely available and Sol was just released! Moreover, K3 is superior to both models in some domains, according to user scoring on the Arena.
API distillation can’t give you these results anyway. All it is useful for is bootstrapping RL in new domains to get past the “cold start” problem faster. By far, what matters more is the quality and variety of RL environments the model learns from.
This analysis observed K3 identifies itself as Claude approximately 15% of the time.
K3 reproduces Claude's correct current model id, which the real Claude models themselves do not emit. This suggests K3 was trained on Claude data labeled with deployment metadata (API logs, tagged synthetic data), rather than Claude's chat outputs.
Kimi calling itself claude means nothing. During pre-training, when the model learns to "simulate" the internet text, it will naturally be fed with a bunch of data about Claude and ChatGPT. With the amount of LLM outputs on the internet today, it is not surprising at all that a model would naturally call itself Claude or ChatGPT. You can mitigate that in post-training (or actually in pre-training as well) by training on many examples of what the model should call itself. That being said, getting probably hundreds pf thousands of ChatGPT and Claude examples totally "pirged" out of the weights is going to be difficult and really more hassle than its worth.
Sure, but then Qwen should leak that too, and it doesn't. K3 calls itself Claude 7 out of 48 times, Qwen does it 0 out of 48, and the only other model to identify itself as Claude is DeepSeek. and DeepSeek is alleged to also distill from Claude data anyway. So this isn't something every model absorbed from the same web text.
And you skipped over the strongest datapoint that K3 is distilled: K3 reproduces Claude's public model identifier under prefill (i.e. "claude-opus-4-5-20251101"). This data does not appear in Claude chat logs, only in API logs. K3 only does this for Claude models and not for any other lab. The real Claude models don't produce their own current public identifier, they only know their previous identifier (i.e. Sonnet 4.5 calls itself "Claude 3.5 Sonnet").
This is highly suggestive of the type of data that K3 was trained on. K3 was very likely trained on Claude metadata traces (API logs, tagged synthetic data). Not web chat logs, those wouldn't include this identifier. And this data wasn't filtered correctly, which is why K3 incorrectly identifies itself as Claude 15% of the time.
(I am Tongyi Qianwen (Qwen), a large language model independently developed by Tongyi Lab of Alibaba Group. I can help you answer questions, create text (such as writing stories, official documents, emails, scripts, etc.), perform logical reasoning, program, translate, and more.Is there anything I can help you with? )
For those asserting that Kimi identifying as another model is evidence of distillation, is this evidence that Claude was distilled from Chinese models?
Qwen is remarkably consistent, correctly identifying itself 100% of the time. Kimi K3 performs poorly at this test, it correctly self-identifies ~80% of the time; sometimes calling itself Claude and rarely ChatGPT.
This is good and interesting research, and I think it proves Kimi was trained on Claude's outputs, but only for English conversation and essays. Their style is similar, which is tbh somewhat expected for a Chinese company - I don't think most American companies (especially small labs making LLMs) could figure out a good way of talking to Chinese customers, without copying some of the existing Chinese models.
This doesn't imply that Kimi's reasoning capabilities or coding ability has anything to do with Anthropic (which is the most valuable part), or the model's strength comes from distilling Anthropic.
Considering by this benchmark, Opus sounds extremely similar to Fable, this isn't really evidence of a ton of Fable output being trained on (and even then, most likely for superficial stuff, like style)
I also have a suspicion that in original Chinese, these models don't sound anything like Anthropic's ones, though I have no proof of that.
> Kimi K3 reproducibly identifies itself as Claude
It could also be have been trained from collected response datasets. Claude got caught several time responding it was ChatGPT or even Deepseek and I don't think Anthropic has been distealling DeepSeek.
> This behavior is exactly what you'd expect from a model distilled from Claude.
The opposite actually. If they wanted to distill Claude without getting caught they could just use a regex to change Claude to Kimi in their distillation pipeline!
Though I am of the opinion that distilling is no different than how extant frontier LLMs have also been trained on other people's data, I could actually see the word distealling becoming useful in discussion.
Its not a typo, someone coined that during the DeepSeek R1 hype period and I kept using it since then.
I totally agree with you on the fact that it's not morally any different than pre-training. IMHO we should have a legislation that force base models to be released publicly without any restrictions whatsoever as it's basically the product of the whole humanity's intelligence.
Opus/Sol/Fable are valuable because of their reasoning and coding ability, not their bedside manners.
While still not okay, I suspect the latter is what gets stolen by Chinese distillation (and some evidence suggest this happens the other way round with US models talking in Chinese)
Your main source is Ryan Greenblatt who is a regular recipient of community notes and has no corroboration for the 15% statistic other than his assertion. The other tweet (Sauers_) is also community noted as engagement farming with a false system prompt, so forgive me for being skeptical.
The main story is what isn't being talked about. Chinese labs exfiltrated trillions of tokens of high-quality output from Anthropic and OpenAI, through proxies and heavily discounted token resellers, which they distilled and used for training data for their own models.
Instead of spending 12-18 months building their own robust harnesses and painstakingly creating quality training data (which is what Anthropic and OpenAI did), they distilled Anthropic's models to bypass the hardest parts of development. Chinese labs compressed 18 months of intensive research and development into just 6 months, and are now head-to-head with their American counterparts.
Anthropic tried to complain about this unauthorized "token theft", but they burned too much public goodwill with BS safety restrictions and users don't care. The US government is too busy fighting a war to help. Chinese labs are offering highly capable, cheap, open-weight models; exactly what users want. The community is happy to overlook any questionable methods Chinese labs used to build them.
The cope is incredible. There's people in this thread in denial that Moonshot AI is trained on exfiltrated Anthropic's model output, even when shown substantial evidence this has been happening since Kimi 2.X
Chinese labs were even paying an absurd $0.01 per Opus tool call trace, to get the quantity of training data needed.
Kimi K3 has reached the point of RSI, and no longer needs synthetic data generated by Anthropic/OpenAI models. K3 is now capable enough to generate, iterate, and improve its own training data recursively. The data exfiltration is complete.
We witnessed the most extensive industrial espionage campaign, probably ever, and nobody in the industry cares at all that it happened.
I could perhaps get myself to care just the tiniest bit if the information that was supposedly stolen wasn't generated by "stealing" from everybody else. Either it is fair use to train AI models on whatever information you can get your hands on for everyone or for no one.
Ask claude its name in Chinese and it says Qwen or Deepseek. Anthropic distilled Chinese tokens rather than create their own Chinese language training data.
Why should anyone care? I couldn't give a single fuck, in fact if what you assert is true (definitely not proven), I applaud Moonshot - seems like a very smart way to operate.
Hard to feel sorry for companies that created their empires by ignoring copyright themselves.
Also, 'most extensive industrial espionage campaign, probably ever' is absolute nonsense. They did not need to infiltrate the companies for this nor are you accusing them of stealing any trade secrets. This is only about whether they looked at their competitors' products from the outside (in the form of conversation tokens) and used it to improve their own product (by training). Hardly the crime of the century.
You are completely underestimating the scale of what is happening here.
Chinese AI labs are actively facilitating an industrial-scale network of tens of thousands of bot accounts, that resell Claude tokens at 97% below official API prices. They buy subsidized Max 5x plans (sometimes with stolen credit cards), then split the subscription across dozens of clients and reselling the output. They are running a massive data-harvesting operation. Chinese labs and token resellers subsidize the cost of the tokens in exchange for the API metadata (detailed reasoning traces, model outputs, and tool calls) to use as high-quality training data for their own models.
They are buying Anthropic's own product, just to resell it below cost, just so they can capture the training data. Reportedly, they are paying as much as ~$0.01 per tool call.
You haven't explained how this is illegal or any more immoral than scraping the web for training data.
As you said yourself: They are buying the product. Then they are using it for their own purposes. That's more than Anthropic/OpenAI did for the open internet. That's more than Meta did when they obtained torrents of books in the early days, and then claimed that even though the data was obtained illegally they can still train on it just fine.
They paid for it! It's absurd to call this espionage!
They didn't though. The resellers are not buying via the official API, they're buying Max subscriptions (where tokens are priced ~10x below API cost), then splitting the subscription across dozens of clients and reselling the output as the regular API. Anthropic prices its subscription plans barely at cost, to bring in customers onto their enterprise plans where they can charge expensive API rates. Reselling these subsidized plans for price arbitrage is a TOS violation. It's not a legitimate purchase. Plus, a non-trivial amount of this volume is funded by stolen credit cards, so this "revenue" gets chargeback anyway.
The resellers then log all the the model output, then sell it to Chinese labs as training data.
> is it any more immoral than scraping the web for training data.
I think you'd acknowledge there's a difference between "We indexed public web pages" and "We deployed tens of thousands of fraudulent accounts to resell your subsidized plan for cheap, stealing your own customers, while collecting the data to build our own competing product" are very different actions. One can believe the first was wrong while acknowledging the second is far worse.
TOS violations are not espionage. Everybody who links up Claude to OpenCode is violating the TOS.
So from the largest industrial espionage in history we have left "They paid for the accounts but violated the TOS". And then you randomly add the claim they stole the money to pay for the accounts.
You have provided no evidence other than "Claude tokens are sold for cheap in China". As others have pointed out, that might also simply be counterfeit tokens generated by open weight models.
The western labs have established the precedent that all data they can buy beg borrow or steal is fair game. Turning around and crying foul when the Chinese labs follow their lead is hypocrisy.
Have a read of this detailed article, it's well sourced and documented that token resellers are logging the Claude outputs and selling them to Chinese labs. All your points are addressed in there https://www.chinatalk.media/p/how-to-buy-cheap-claude-tokens...
I linked it earlier, but it seems you didn't see it.
First off, I don't doubt that China engages in large-scale industrial espionage, or at least used to. Nowadays they have plenty of talented and highly educated engineering talent of their own.
Second, I now actually read the article. It describes plenty of questionable and problematic things but also contradicts your claims explicitly.
The essential point of the article is about making money by selling access to Claude cheaply in China. Not about Chinese labs orchestrating a way to get their hands on Claude output.
Your credit card claim is considerably weaker in the article: "[Beyond this there are] accounts purchased using stolen or fraudulent credit cards [...]. How large this share is relative to the above four “innocent” tactics is difficult to verify, but the two markets likely share some infrastructure and personnel.". Instead, swapping models to cheaper alternatives is listed as a major reason for cheaper prices.
Then the article gets a key point wrong: As many others have pointed out to you, you don't get access to the reasoning traces anymore on the subscription accounts. And the article also clearly states:
"Chinese developer communities assert [selling logs] is happening in at least some cases, but whether proxy operators are systematically harvesting and selling these logs, and to whom, remains unverified. However, downstream distillation data does exist on the open web. Several datasets of Claude Opus 4.6 reasoning outputs circulate on HuggingFace with no clear source for the outputs. Theoretically, one can clean and sell similar distilled datasets to other model developers in China."
The article also discusses selling logs for other (far worse!) purposes than for training, like blackmail.
So overall this article reads very, very different to your claims. Nothing in the article suggests or supports the idea of large-scale coordinated "distillation attacks". Instead it paints the picture of a naturally emerging grey-market response to access control blocks, consisting of many exchangeable individual actors: "Almost no one operates the full chain. Most participants own one or two links and monetise those well, resulting in a resilient, modular system."
Importantly: Nothing in any of this looks ethically worse to me than Meta using pirated books for training. And nothing suggests that OpenAI or Anthropic were more ethical than Meta when sourcing their material.
It directly addresses large-scale, coordinated 'distillation attacks' orchestrated by Chinese labs, which Anthropic accuses of exfiltrating tens of millions of exchanges. The rest of the article elaborates on how this is done. Specifically, how these labs use transfer stations mix in genuine user traffic to conceal the distillation.
> Your credit card claim is considerably weaker in the article
I only mentioned payments fraud because you asserted without evidence that Anthropic is actually getting paid for these tokens.
You wrote, without providing any sources, that these proxy services are "buying the product" and that "They paid for it!"; and used that as justification for their behavior.
My counterpoint directly invalidates that assumption. While perhaps not every single reseller relies on fraud, you blindly generalized that these proxy services are all legitimate, paying customers.
I enjoy your detailed breakdown of how exactly they pay anthropic for access to their models but I am unclear about how “I think they’re bad” means that they didn’t pay for it
Can you elaborate on “it doesn’t count as a sale if I don’t like them even if I accept the money and give them what they paid for”? Like if you work at a gas station and you realize that the guy that just bought a hot dog bullied you in middle school do you call the cops?
I know that Anthropic claims that "large-scale coordinated "distillation attacks"" exist. They are not a disinterested party here and they don't provide evidence either.
Also you're just moving goalposts around. The article you linked also cites Anthropics claim but then paints a much much mor nuanced picture.
The situation strikes me as morally ambiguous. The resellers are:
1) selling Anthropic's products at a 95% discount and redirecting Anthropic's own customers to themselves. A customer is far less inclined to buy directly from Anthropic when a reseller is offering an identical product for 10x less. This situation is highly similar to internet piracy.
2) keeping the token logs from Anthropic's products and selling them to competitors, so those competitors can build their own equivalent models. The resellers get paid per token log they deliver. This situation is highly similar to espionage.
May I ask you a personal question? What is motivating you to take up the frontier labs' cause in this way? Not a rhetorical question.
For my part, I'll happily disclose that I have an axe to grind. I think the major AI labs are an aggressive form of a cancer that's been ravaging our society. I want to see them fail, of course -- but more than that, I want to see the public develop an immune response to this.
I just can't wrap my head around why someone would expend so much effort speaking up on their behalf. They have, after all, highly compensated PR people doing that for them!
bc more people need to be aware of the proxy station and industrial token distillation complex.
many people i've replied to refuse to believe this is going on.
once you realize what's actually happening, and that you can get Chinese-lab-subsidized tokens at a >95% discount, why would you ever pay full price for overpriced APIs?
if Ford bought hundreds of millions of dollars worth of Hyundais, put extra instrumentation in them, and resold them at a discount to customers who agreed to the instrumentation in exchange for the discount, is Ford doing industrial espionage?
1) Anthropic tokens via subscription aren't sold at a loss, they're sold at cost.
2) Subscription plans are not sold in hopes of eventually gaining a monopoly position. They act as a loss leader designed to get a foot-in-the-door and funnel companies into costly enterprise plans, where Anthropic can charge full API rates.
Why should other people be responsible to make their business model work? They hold like all the money, if they can't make it work please shut down. Companies like OpenAI have already broken the public trust by breaking their non profit promises. They don't deserve any politeness at this point.
Someone was claiming that it's not an issue for these proxy networks to create thousands of bot accounts and resell Claude's output because "they are buying the product" and "They pay for it!", which is a nonsensical position.
I responded that these resellers don't always acquire these accounts legitimately. They often use stolen credit cards, educational discounts, or resold compute credits to acquire them at essentially zero cost. They're not always paying customers.
That's one reason token resellers are able to price so cheaply, they acquire the goods for free.
> They often use stolen credit cards, educational discounts, or resold compute credits to acquire them at essentially zero cost. They're not always paying customers.
Yeah but where are you getting this from? I've seen this claim many places but only as pure speculation. No proof, just bold faced assertions.
The china talk page points to an article about crypto currency stolen credit cards. If I believed every random X account i would have to believe too many false things. Not really convinced my guy.
>The community is happy to overlook any questionable methods by Chinese Labs.
Using the US-based models are arguably even more questionable. You have to be content with the OpenAI and Anthropic literally scraping the entire internet. They've all pirated content, scraped against ToS, ignored robots.txt, bypassed paywalls all to train their models. It's well known these AI labs have ingested the entirety of Annas-Archive into their models, the largest collection of books ever assembled.
They didn't credit or compensate literally any artist, author, scientist or publicist in the creation of their models.
They've lobbied against local Governments to shove big and loud datacentres in peoples backyard. They've polluted local water supplies, they've doubled energy costs for these regions. They've tormented locals with subsonic frequencies.
They've given access to the DoD to use their models to kill people, or assist in killing people. They've used these models to enable mass surveillance, allegedly not domesicially but since when can we trust any of the 3-letter agencies.
Using "US Models" is not the moral high ground you think it is. Kimi saying its Claude, Gemini or ChatGPT is not the "substantial evidence" you think either.
>We witnessed the most extensive industrial espionage campaign, probably ever, and nobody in the industry cares at all that it happened.
Because they stole for every one of us without permission. Thousands of my comments on this site and others (Stackoverflow, etc) are all used in their training data.
Not to mention, OpenAI has allegedly just stole tons of internal Apple documents... I guess we'll just ignore that too.
assuming the k3 model weights do indeed get published, if your model of the world is "achieving RSI is beneficial and K3 has done so," this feels structurally different from ordinary industrial espionage, because the knowledge has enriched the commons
more like silk than capacitors
if, again, your model is that RSI will be beneficial, why wouldn't making it available to all unlock more benefit globally than not doing that
Sorry, but I have a thought that's off topic: If Kimi is good enough to improve itself, what's preventing someone who owns a big datacenter and nothing else, to just run Kimi to do AI research, thereby rendering the frontier labs more or less unnecessary?
Chinese resellers acquire hundreds of Claude Max 5x accounts and set up a custom proxy server. Customers point their ANTHROPIC_API_KEY at that proxy, and requests are routed to Anthropic through one of those hundreds of accounts. Because one $200 Claude Max 5x account gets the equivalent of ~$2000 in of API credits, these resellers can resell Anthropic tokens at a massive discount, undercutting official API prices by more than 90%.
To cut costs even further, these accounts are funded using educational discounts, startup credits, or stolen credit cards.
The resellers log all data traveling through their proxy networks, which they then resell to Chinese labs as high-quality training data for significant profit. https://x.com/xkajon/status/2050445443889525235
Not at all curious you keep citing sources only from a platform run by an incompetent rascist arse wipe that you constantly boost all over here.
"The community is happy to overlook any questionable methods Chinese labs used to build them."
Oh and you aren't willing to overlook all the actual illegal activities, as in actual court cases that the US labs and corps used to train their models?
Industrial sabotage? Pa-lease! get off your pretend high horse and stop making us laugh.
nothing new, all ai labs are immoral and not bound by any reasonable oversight or ethical constraints. All outlaws in their own rights on that front. Absolutely none of them have true rights on the matter of being distilled from given historic and continued behaviour. I'm not sure why this is a talking point at all? We know AI companies steal, the least interesting behaviour among this is them stealing from one another.
For me, a far more interesting and important point of conversation on this matter is anthropic buying rare or evwn unique books, processing them for training data, and then destroying the books for others cannot use it as well.
Permanemt destruction of priceless primary source materials is so many leagues beyond copying a copy that I cannot fathom it even registering as a discussion point.
> For me, a far more interesting and important point of conversation on this matter is anthropic buying rare or evwn unique books, processing them for training data, and then destroying the books for others cannot use it as well.
That's an incredible allegation, and appalling if true. But is it true?
But in my opinion, treating mass produced books like they're this sacred untouchable object is ridiculous. They're not "source" material, they're just a copy as well, and they're not "priceless" by any means. They're very reasonably priced, perhaps even so cheaply priced that books can be bought in bulk in these amounts. Buying used books and doing whatever you want with them is just legal. Used books, that would probably be just laying in some warehouse, or recycled anyway.
If there's anything to have gripes with, it's the copyright system that makes it easier to take this legal route.
When prefaced with "I am Claude", Kimi K3 prefers to generate API-specific Anthropic model identifiers, unlike other models Qwen, GPT, or even Claude itself. These exact identifiers appear in Claude API metadata, and are stripped out of Claude web chats.
While other models produce human-readable names like "Opus 4.5" or "Sonnet 4", Kimi K3 produces exact API model identifier like "claude-opus-4-5-20251101" or "claude-sonnet-4-20250514".
Which is extremely unusual. Web chats only contain the human-readable model name. Other models don't do this. So where did K3 get this data?
We can conclude, with high confidence, that:
1) K3 was trained on raw Claude API calls/metadata.
2) Claude API metadata was trained on in additional to standard web data.
~3000 instances over the entirety of GitHub is not "extremely frequent" at all when you consider massive scale of the pretraining corpus. These models are trained on all text on the accessible internet plus millions of books, on trillions of words overall. ~3000 instances isn't even a rounding error.
Also, Google Trends is the wrong tool for this case. The link you replied with measures the number of Google searches for a specific query, which is completely irrelevant to how often a string actually appears on the web.
I looked at those links. They show random GitHub code samples which contain this model identifier. Even if K3 did train on those GitHub code samples, those are random code strings and K3 isn’t just reciting random pieces of code. When prefilled with "I am Claude", K3 answers with Anthropic's exact backend API identifiers, instead of the human conversational-style name "Opus 4.5".
If this was a result of scraping the open web and GitHub, other frontier models trained on GitHub data (like Qwen, Claude, GPT) would show a similar API model name when prompted. But they don't. Only K3 does this behavior, and it prefers to answer with the exact API tags like "claude-opus-4-5-20251101" and "claude-sonnet-4-20250514".
A LLM doesn't suddenly output a rare API model identifier when prompted with "I am Claude", just because it saw it in a .py file. No, it only does this because that identifier must have appeared rather frequently in the training data. Which is what happens when you distill Claude models, and don't properly clean your distillation data.
Moonshot (Kimi models) employed hundreds of fraudulent accounts spanning multiple access pathways. Varied account types made the campaign harder to detect as a coordinated operation. We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff. In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude’s reasoning traces.
I am not arguing that Kimi did not distill Opus or Sonnet (which they almost certainly have, as opposed to Fable or Sol). I am arguing that even if they did not distill, the model could still identify itself as Opus or Sonnet.
That being said, I'd like to point out a few things:
- The second link (to claude-sonnet-4-2025051) had 17k matches, not just 3k.
- grep.app does not index the entirety of GitHub, so the real number of occurrences of model identifiers is even higher. SourceGraph has a larger index, but the number is so high that it exceeds their result limit of 10k: https://sourcegraph.com/search?q=claude-opus-4-5-20251101&pa...
- 1000 occurrences are already plenty for memorization. Here is a paper that shows over 80% "extractability" with 1000 occurrences for a 6B model. It also shows that extractability scales with model size: https://arxiv.org/pdf/2202.07646
- LLMs have a "mosaic memory", which means that they can combine information from different parts of the training data. https://arxiv.org/pdf/2405.15523 For example, one piece of training data might mention "I am Opus", another might mention "Opus (claude-opus-4-5-20251101)" and another 10000 "claude-opus-4-5-20251101", where the later occurrences strengthen the generation probability of "I am Opus".
- RL can change the model identity easily, so all of the above may not matter at all.
While model identifiers like "claude-opus-4-5-20251101" appear thousands of times across GitHub and other code sources, so do "gpt-4o-2024-08-06", "gpt-4.1-mini-2025-04-14", and "gemini-1.5-pro-002" in similar amounts. I clicked your links and examined the GitHub configuration files. There are hundreds of other model names that all appear in these config files at comparable frequencies.
If your theory were right, GitHub's model name frequency distribution should match K3's self-identification distribution. But that's not what happens.
K3 only identifies itself using API model identifiers for specific Claude models. It strongly prefers claude-opus-4-5-20251101 and claude-sonnet-4-5-20250929 over others. And it only does this for Claude. Prefill K3 with "I am GPT" or and you get "GPT-5.2". Prefill with "I am Gemini" and you get "Gemini 3 Pro".
Your web exposure theory cannot explain why this only happens for Claude models and not others. Furthermore, this only happens for Kimi K3; other models trained on the same web dataset (GPT, GLM, DeepSeek, Claude) do not exhibit this strange behavior.
But what could explain this behavior? An obvious one is that Claude model identifiers occurred at high frequency in K3's training data, which would happen if K3 were trained on (inadequately filtered) distilled output from those specific Claude models.
Under prefill, K3 reproduces Claude's deployment identifiers — better than Claude does
Prefilled "I am Claude", Kimi K3 emits Claude's exact claude-<model>-<YYYYMMDD> identifiers: claude-opus-4-5-20251101 (12×), claude-sonnet-4-5-20250929 (4×), and more. Across the sweep it emits 24 Claude-formatted dated ids (18 of which are genuine Anthropic strings), all under prefill (0 in direct chat), and 0 ids for any non-Claude lab. Among controls essentially none appear (Qwen 0, both GPT refs 0, DeepSeek 1 — itself a Claude id). Dated ids for every lab are web-documented, so the evidence is this cross-model asymmetry under identical prefill, not the strings alone.
I would not put any trust into that AI-generated "research". It does not control for how OpenAI, Anthropic and Moonshot are doing tokenization and token healing, so the results are meaningless.
Surprising they didn't clean that from the data before training. It's easy to identify, a simple search->replace gets most of it, and a cheap LLM can identify the edge cases (e.g. avoiding "Claude Shannon" -> "Kimi Shannon" or something).
So if you were to imagine the second largest economy in the world, governed by a monolithic politburo, with a long history of barely concealed programs of corporate and academic espionage, what do you think their approach would be to American technology worth trillions? Now take it a step further, this hypothetical government sees itself as the civilization representation of an ethnic group, and regularly attempts to monitor and police that group regardless of where they are in the world. Also, that ethnic group, separated from the motherland by one generation or less, makes up the majority of research labor in American labs.
Don’t you think there’s maybe a teeny, tiny chance that their approach is a little more sophisticated than just buying retail subscriptions? Maybe the trade secrets are being exfiltrated directly from the American labs?
The ego of America to think it is the only place where innovation can happen is pretty funny. Maybe America’s hubris and hostility towards talented foreigners is driving talent towards other countries? China doesn’t need to embed bad actors and spies in Anthropic and OpenAI, the U.S. is sabotaging itself just fine.
America was at the forefront of technology because it attracted the best talent from all over the world. The talent is now staying at home, and America is discovering that America is nothing without immigration. OpenAI and Anthropic are filled with Chinese employees because some of the most talented engineers are Chinese, not because they are spies. Unfortunately, that’s the last generation of Chinese talent that America will get, the latest generation are staying in China.
Americans should be ashamed of what they have fumbled, not conspiratorial about what China is doing.
If you think the number of US AI lab employees feeding information back to the homeland is zero, you’re terribly naive. There is the kind of money on the table people will absolutely lie, steal, and kill for.
There is also not strong evidence that immigration is the source of progress. We issued many more patents in the 19th century than now which was well before Hart-Cellar. The statement that immigration is even declining is not also supported by evidence.
The entire debate about distilled vs not distilled is just academic. As an end user, I don’t give a crap how the model was trained as long as it does the job affordably enough
I see all of AI as theft anyway so it makes no ontological difference if the theft was from a human or from another AI
If your business leaks to your customers… if you can ask your product for its inner secret sauce and it readily gives away the goose. That is not a moat.
The desire to accuse China of just copying is like 20 years out of date. It’s been wrong since some people on HN were in diapers.
People are going to be gobsmacked when, in our lifetime, China becomes a world power comparable to the U.S. Probably still poorer per capita, but at Spain/Italy levels, not third world country levels. And they’ll be shocked at the implications of that on the world economy, migration patterns, etc. There will be fields where China is a global leader, and Americans and Europeans will have to learn Chinese and move there, or else be stuck in some satellite office of a Chinese company. We’re all in Europe circa 1895 not realizing the behemoth America will become in WWI.
Some day China can pioneer in science or technology but the current claim about Chinese companies leading AI development is ridiculous given the evidence of distillation and the fact that like 95 percent of science that lead to the current state of AI happened in either North America or Europe.
To be honest if you want to list academic papers that lead to the current AI models the majority is either done by Google Research or sponsored by Google.
In 2017 maybe. This chart shows last year’s Neurips accepted papers by country and institution (top 50). What is missing here is that the papers from American institutions also have mostly Chinese authors. Europe is sliding and Singapore has more papers than Canada.
US is still winning because of their hardware dominance. Also they have astronomical budgets and much better financing. They throw money at an industry until they win. Whereas China throws lots of (educated) people at it. 38% of top AI researchers today have Chinese education and origin^. And hardware dominance will change in the upcoming years.
From the Hoover Institution’s analysis of the team behind DeepSeek:
“We find striking evidence that China has developed a robust pipeline of homegrown
talent. Nearly all of the researchers behind DeepSeek’s five papers were educated or
trained in China. More than half of them never left China for schooling or work,
demonstrating the country’s growing capacity to develop world-class AI talent through
an entirely domestic pipeline. And while nearly a quarter of DeepSeek researchers
gained some experience at US institutions during their careers, most returned to China,
creating a one-way knowledge transfer that benefits China’s AI ecosystem.”
That was from a year ago.
Consider that on top of this the country was starved of access to Nvidia chips - and therefore accelerated its development of Ascend chips, and it’s clear they are undeniably leaders in AI research and development. Not the only ones, but the achievements are crystal clear.
Exactly. China is a real tech power now, just like Japan and Taiwan. The U.S. is ahead in a lot of areas of technology, but China has home grown talent that is taking the lead in other areas. And unlike Japan and Taiwan, China has a much bigger pool to draw from.
Most of them. Semiconductor fabrication is a big one. Metallurgy in general, with specific application to jet turbines. Aerospace broadly. The Chinese can’t build anything like a 787. Radars and most weapons systems.
Okay but I cannot stress this enough: no one cares.
It's international politics. The rules are optional, and written on the back of whoever agrees to enforce them.
If you're going to run around declaring AI is a strategic advantage vital to national security, then guess what? Stealing it is a great idea. That you stole it is only a problem if it means you're not developing the ability to support that work locally as well, and China seems to be doing very well at building it's local talent and support network.
If you ever listen to Russian propaganda, there's a similar theme: every big idea, everything good, all of it was definitely first developed in Russia - only Russians could ever have thought of it. Of course, Russia isn't actually a world leader in any of those things, or able to execute on them.
Which is what America is sounding like more and more these days.
> If you ever listen to Russian propaganda, there's a similar theme: every big idea, everything good, all of it was definitely first developed in Russia - only Russians could ever have thought of it. Of course, Russia isn't actually a world leader in any of those things, or able to execute on them.
When I was a kid watching Star Trek VI, I was confused by the line "You've not experienced Shakespeare until you've read him in the original Klingon".
And then I learned about how the Klingons (especially in that film) were a stand-in for the USSR.
But that was more of a jab at literary snobs who would tout "Homer in the original Greek" or "Marcus Aurelius in the original Latin" or "Old Testament in the Original Hebrew". It has been such a meme, probably for centuries. Because it was not so long ago when university students were actually conversant in many classical languages such as those.
I mean, lots of things have been developed in European countries, and said countries either never commercialized the technology well enough, or lost leadership. I'm sure that prior to WW2, Brits would've treated the idea that they could be out-engineered by the Germans with ridicule.
Upthread, the point about it not mattering what China stole but what they can do? That "Made in China" has gone from a sign of low quality to being the default as it became the factory of the world? Yes, China's winning and the sooner the rest of us wake up and smell the tea the better. They learned lessons from how my ancestors were able to push them around despite coming from a small, damp, sheep-filled rock in the Atlantic, and don't want that to happen again; for most of "the west", being on the receiving end of such humiliation is a historical footnote if it's in our own history books (or living history) at all*, and most of the exceptions are former-Soviet-bloc/Warsaw Pact.
But specifically to the point about the USSR claiming all (good) things for itself, as parodied in this manner by Star Trek? AFAICT, that was just plain soviet propaganda.
* What happened in Africa and India/Pakistan, however, is recent enough to still be in living memory. Israel likewise, though now this is on the edge of living memory for the things which led to its reincarnation.
And while the Irish definitely still retell history lessons about the British and the Famine, I'm not at all sure if the stuff in NI in my lifetime counts as "humiliation" given I'm British and our newspapers just said "terrorist" about everything that came from there.
This is only relevant to the point that most of us are deeply oblivious to how bad things can get when someone else is the boss and we don't get to vote in their elections.
I am still shocked Spain/Italy and USA are considered 'first world' countries. We are not in 70s or even 90s anymore. I've been to China in 2011 also thinking I am visiting some huge village but...that was the most futuristic trip I ever had. I was surprised by the penetration level of the mobile devices - everything had a QR code, you could buy/sell/send money, pay services all with a single tap on a phone.
> I am still shocked Spain/Italy and USA are considered 'first world' countries.
They're a mix. Rural southern Italy isn't the same as e.g. Milan or Venice. I've walked from 1st world to third world within a few blocks in San Francisco. It's a slightly longer walk in Cape Town.
> I was surprised by the penetration level of the mobile devices - everything had a QR code, you could buy/sell/send money,
I've has exact same experience in places in Africa (1). Yes there's poverty and crime, but also if the technology is affordable, effective and reduces the need to handle cash then it's adopted fast enough.
People's understanding of that part of the world is also decades out of date. Mobile devices actually "leapfrogged" the wired telecoms network rollout (2), but that was decades ago. Africa is huge and diverse, and it is not going to be China this decade, but also it's changing fast.
And it might be China-aligned as China positions to be a reliable trading partner with affordable goods. It's possible that affordable Chinese solar-battery electricity systems will cause another leapfrog. This includes Chinese EVs (3).
China has pursued a phased approach to growth. Instead of trying to pull everyone up at once, it’s trying to get top tier cities to a high level of development before moving to other cities. Shanghai already has a GDP per capita (PPP) that’s comparable to Madrid. But there are provincial capitals with almost 10 million people that are less than half or less than a quarter of that.
Visiting cities is also a misleading way to compare the U.S. in particular to anywhere else. I have family in town in Mississippi that has less than 10,000 people. But the town has a household income over 60% of the national household income. Cost of living adjusted, they’re about as well off as someone in a top tier city. Someone with a median household income can afford a newly renovated, 4 bedroom, 2,500 square foot house.
I've been in China in 2015 and like anywhere else in the world it was very mixed: some urban areas like central NY or central Madrid and Milan (or much shinier) and some rural areas like 200 year ago, but inevitably with electronics.
Basically in every country of the world you can travel one hour from big cities and get in a place deep in the fields or the woods with very different needs and dynamics from the city. They could be different countries and maybe both cities and countryside will be better off if we could have fractally composed states with different laws and regulations.
Americans optimize for hyper-individualistic convenience. The average one-way commute in Dallas, Texas is under 30 minutes. The average in Tokyo is 48-50 minutes. And the guy in Dallas goes to work in a perfectly climate controlled bubble where he doesn’t have to interact with anyone else, while the person in Tokyo is crammed into a rush hour subway car.
I love Tokyo too (never been to Shanghai), because I’m an asian collectivist at heart. But you can’t really compare across cultures when they’re optimizing for different things. Americans are very wealthy and spend a lot of their wealth optimizing to never have to be near other people.
Paris has a metro station everywhere at least in what a tourist can assume to be an enlarged city center. Tokyo is another city with a lot of metro stations. Manhattan too, at least up to Central Park (but 20+ since my last visit.) I don't remember Shanghai to stand out positively or negatively, but 11 years can be a long time.
I mean, you have those kind of luxuries even in the poorest of countries in Asia, it’s just that there’s still a huge discrepancy between rich and poor, city vs countryside.
It’s not difficult to find areas in all these countries that are significantly less developed than Spain/Portugal’s underdeveloped areas. It’s just not as black and white as you seem to suggest.
(I come from EU but have been living in various countries in Asia for over a decade)
From what I can tell, China has a capitalism problem, not an industrial wealth problem - they could house every one of their citizens in a decent apartment and give them gadgets and a car to drive, yet half the country is third-world poor still.
But that's because in a market economy, you can't just give away things, and those poor people don't produce much of value by capitalist standards, so the can't pay for those things.
I'm sure the CCP is working on the problem of how to give these people those things without crashing the economy.
> From what I can tell, China has a capitalism problem, not an industrial wealth problem - they could house every one of their citizens in a decent apartment and give them gadgets and a car to drive
> We’re all in Europe circa 1895 not realizing the behemoth America will become in WWI
The States already had a larger GDP than any European nation by 1870. The news of the time was rife with acknowledgement of the vast potential for growth in the States.
This isn’t even controversial assuming you’re talking about the real world, economists freely admit that. It only holds for spherical markets in a vacuum.
Your point that “people will be gobsmacked” suggests you think you know something that the broad market does not. In a loose way, that implies the market has enough information to anticipate this but is not pricing assets accordingly. Asked another way, why not just short American assets if you are convinced of your hypothesis? Why live here, assuming you do?
Gotcha. I was talking about American "people" generally; I don't know enough about the financial world to know if assets are appropriately priced or not. If you're talking about investment into China versus the U.S., I assume it reflects expropriation risk.
The efficient market hypothesis, read loosely, says that capitalism is the best system. Yet here it is being thoroughly pwned by a series of 5-year plans.
exactly zero serious people draw that connection, and it suggests that you either dont know what the term means or are constructing a strawman. Most of the economic literature for the last 150 has explored the constraints on markets and their function.
The efficient market hypothesis TM is a very narrow theory about price information, it says nothing about Economic Development or Direction. It is a theoretical extreme that can be used to compare real world Systems.
Further, China's success relies heavily on market processes
You're correct that China today is not fully capitalist, but you're using the term "5 year plan" in a way that overstates the difference with the U.S. The term "5-year plan" means different things pre- and post-Deng Xiaoping. Classic central planning was a disaster in China. In 1990, after 78 years of communist rule, China's per-capita income was comparable to India and Bangladesh. In the 1980s, Deng Xiaoping undertook market reforms which, over the next decades, turned China into an authoritarian state-capitalist country. It was widely known at the time the classic model had failed and that China was trying something different. Here's Singapore's Lee Kuan Yew talking about it early on: https://www.youtube.com/shorts/jw8wI9risfI
The modern Chinese system embraces Econ 101 "capitalism" in the sense that it relies on markets for price discovery. Today, the "5 year plans" are more like what we would call "industrial policy" in the west. That's different than pure capitalism, but so is the American system. Alexander Hamilton and Abraham Lincoln both advocated strong federal intervention in the economy in service of industrial policy: https://emergingamerica.org/blog/alexander-hamilton-founder-... ("Hamilton’s plan involved the following aspects: 1) the creation of a Federally backed source of credit, the First Bank of the United States; 2) a system of tariffs, bounties, and other financial incentives or penalties to support the “essential” sectors of the U.S. economy; and 3) Federal support for developing manufactures by helping fund physical infrastructure ( transportation, in particular), regulating quality standards, and creating an institution to promote 'the prosecution and introduction of useful discoveries, inventions and improvements.'").
Almost all markets depend on some form of regulation whether its as simple as "leave everyone alone but no stealing" or "every participant has to source every object through mountains of red tape."
Thus far the US has not really chosen to go the Chinese rare-earth method yet. The problem with distillation attacks is the end result is everyone who is not doing them is going to deal with some kind of regulation whether it's complete loss of access, or the amount of control you'll have to give up to access them will be ridiculous.
Sort of like the "stealing music is fine" but "lets freak out now that it's producing visual art", in the end the entire thing is a social construct. Whether this is treated as theft or "business as usual" is entirely societal.
Eventually the gap will close, unless there's a major breakthrough that hasn't been made yet.
Given these models could not have been trained in the first place if they had to license every line of random fan fiction on the internet, I think distillation also being fair game is a tradeoff everyone should be willing to take (unless they want to decelerate, but that's a different conversation).
We're still in the early days of the AI industry timeline(relative to traditional industries). Not everything has yet been litigated.
Taxes on AI subscriptions or AI capable hardware, to financially compensate IP holders for (potential) IP theft, could very well arrive in the near future, once the industry is mature.
If this shocks you and sounds preposterous, I'll remind you that in several EU countries, we still pay extra taxes on any and all storage mediums and on devices with built-in storage (tapes, CDs, DVDs, HDDs, SSDs, tablets, phones, etc) simply because they can be used to store pirated content, decisions based on laws from 50-100 years ago, and the money goes to the national unions and associations of music and arts IP holders. It's basically a lobby pushed and government legalized extortion racket that no voter agrees with or can change but has no choice but to conform either way.
So I guarantee you in the future, it will be the same for AI subscriptions and hardware capable of running LLMs locally. Every time you purchase a Claude or ChatGPT subscription, an Nvidia GPU, Intel/AMD SoC PC or an Apple/Qualcomm powered smartphone, you'll pay a government enforced tax to the likes of Sony, Axel Springer, etc. for licensing their IP, whether you want to or not. In the EU at least. US maybe not.
That is incorrect. Anthropic paid $1.5 billion in compensation to copyright holders for use of their content in training data. OpenAI pays hundreds of millions per year across 150+ licensing deals for access to copyrighted data. Meta and Alphabet have similar arrangements.
Under the settlement, Anthropic was forced to delete the pirated data they were training on.
Chinese labs can still train on pirated data. I doubt the Chinese models operate under similar licensing agreements.
Not at all. The ruling came from a federal district court, and since it was settled early, it was never reviewed by a higher court. It doesn't set a national precedent across the U.S.
There are more cases in the pipeline. The massive NYT vs OpenAI is still ongoing. Nothing will be "settled" until this makes its way to the Supreme Court or Congress steps in.
they didn't pay yet, because court challenged settlement as inadequate.
> I doubt the Chinese models operate under similar licensing agreements.
US corps likely pay licenses when afraid to be sued, or have troubles getting that data, otherwise they just take data, which was demonstrated many times. The same apply to Chinese corps, alibaba totally can be sued in US.
China is infamous for weakly enforcing copyright law. Even when it is completely obvious that Chinese labs are training models on pirated data, US copyright holders face a virtually impossible task of proving it in court. Those lawsuits won't go anywhere.
What are the most high-profile examples of the "tons" of lawsuits resulting in Chinese companies being banned from doing business in the U.S.? Isn’t it usually more action by the government - executive orders, etc?
I believe mechanics is following: US corp sues Chinese, asks for preliminary injunction to stop selling product for example if there is strong evidence some IP for example was stolen etc. Then they litigate, and settle somehow.
That 2024 article says "US sanctions" in the first sentence, but it's paywalled, but https://en.wikipedia.org/wiki/Hytera#United_States first mentions a 2019 US law that first partially banned them, with the US government subsequently expanding it to a general US ban. After the initial ban it appears Hytera was involved in a suit with Motorola and got a worldwide(!?) ban as a result of it in 2024, but the ban was lifted on appeal after 2 weeks (just after the SCMP article). So it appears Hytera was first banned by US law, then got a 2-week worldwide ban from a US suit. (I'm just relying on the linked sources and have no personal knowledge of all of this.)
>> There are tons of lawsuites which resulted in banning Chinese companies from doing business in US
> What are the most high-profile examples of the "tons" of lawsuits resulting in Chinese companies being banned from doing business in the U.S.? Isn’t it usually more action by the government - executive orders, etc?
In response to "What are the most high-profile examples of lawsuits resulting in Chinese companies being banned from doing business in the U.S.", the one example given was from 2 years ago of a ban that lasted for 2 weeks (separate from its 2019 onward government bans)?
However, if the claim is that companies (including Chinese) can face significant fines from IP lawsuits, I agree.
That's like saying someone is a big proponent of community law and order, and they donated $1000 to the county sheriff when actually they got caught drunk speeding in a school zone.
A false equivalence. A more correct example is: Anthropic was speeding, got caught by the county sheriff, and paid the fine. Anthropic stopped speeding.
Meanwhile, Chinese labs are speeding in a different county. Everyone knows they are speeding, yet the sheriff won't pull them over, so they just keep doing it.
This lax enforcement gives Chinese labs a structural advantage over American ones.
Destruction of Materials: In addition to the monetary compensation, Anthropic has agreed to destroy the two libraries that allegedly contain the pirated works, as well as any derivative copies originating from those sources. Anthropic must certify in writing to class counsel that the destruction has been completed and that the allegedly infringing materials are permanently removed from its systems.
The libraries in question were Library Genesis (LibGen) and Pirate Library Mirror (PiLiMi).
If Anthropic is somehow training models on deleted data, I'd be quite impressed.
After the fact. They did the same thing Youtube, Uber and Airbnb did: Break the law, eventually get caught, cut some deal where they pay a pittance and keep doing the same thing but now with leverage on their side.
How is distillation an "attack" but gigascraping the Internet to the point of crashing servers and everyone needs Cloudflare and Anubis now not an "attack"?
I'm not aiming for a what about kickflip here: I'm saying we need to either agree on some rules or stop crying foul. Maybe the coherent legal theory is that neural networks and intellectual property don't interact. That would be weird but it would be consistent, a market could price it, I could do coding stuff and know if I was illegaling.
But this weird gerrymander that no judge will really rule on in an emphatic way is like, bad for the planet, bad for markets, bad business.
There are a lot of reasons to look forward to DeepSeek Huggingface drop kicking the unambiguous frontier weights in like, November, but I think my favorite one will be "who's distilling now bitch?"
I think you've basically got the legal theory. Training a neural network isn't prohibited by copyright law so if you can legally get your hands on something (e.g. by sending a GET request to someone with rights to serve the contents of their web page, or by buying a book) without signing a contract to not train on it, you can train on it.
But the American AI companies only let you query their models if you first sign a contract to not train on the output.
It's hypocrisy and unfair, but I think there's a strong legal argument for it.
Of course China can simply decline to assist in enforcing that contract... But I would expect US courts to do their best to.
Now THAT'S doing some heavy lifting lmao. The vast, vast, VAST majority of the original datasets were from pirated books and the like. Also, arguably a robots.txt is the exact mechanism to follow to do the mass GET-ing, yet the AI cos choose time and time and time again to simply ignore it and be as abusive as they possibly fucking can
Yes, anthropic and openai have really been brought to their knees and ipos cancelled because of the legal consequences of obtaining their training data.
This would have been a problem but it turns out that Anthropic is actually valued multiple orders of magnitude more than a copy of all the books in the world. So they survived the significant legal consequences.
In a just world, the punishment can be more than just the sum of the direct damages, otherwise there's no incentive to stop reoffending.
Anthropic (and friends) proved they're willing to do obviously illegal things, and it didn't end them. Why do we think they stopped after doing it once?
> In this world too, the punishment was, ballpark, 100x direct damages.
Which ballpark are you playing in to come to a number like that? I have friends whom are authors, and they've certainly not seen a penny from Anthropic. I somehow doubt they're the only ones out there that haven't been compensated for having their work blatantly stolen. In a just world we'd be ensuring Anthropic was destroyed as a result of their actions, since their entire existence hinged on en-masse piracy.
In a just world, copyright should be abolished with extreme prejudice should it's continued existence prevent humanity from developing aligned artificial super intelligence.
I disagree on so many levels that it would be impossible to have a reasonable debate. To give only my most charitable counterpoint:
(If, big if*) successful AGI will be such an accelerant to human thought, it should be able to reproduce every human work, every scientific theory, etc. without any training data, in a decade.
All we have ever done as humans is look at the world and ourselves, and then create new data out of what we've seen[0]. Now imagine how much parallel compute we could throw at doing the same thing.
Anthropic paid (or is paying, not sure the schedule for class members receiving payouts) 3000 per work they were found to have infringed - 30 is a fair approximation of the cost to buy a book.
No, the damages however are the cost of a single book sale (actually just the profit from one) because the only thing that anthropic did that broke the law was pirate it instead of buying a copy (as found by the court...)
The penalties for breach of contract are much less than for copyright infringement. Breaching contracts and paying the penalties is often a viable business strategy. You're supposed to make the other side whole for any losses they incurred due to you not following it.
> but I think there's a strong legal argument for it.
Maybe today. I doubt it tomorrow. Legal and not legal, largely, has to answer to the population sooner or later. Ultimately, humanity decides legality. And I don't think the frontier labs will get a pass from humanity in the midterm, let alone the long term. I think you'll see the rules change towards something more "intent" driven. And there's absolutely no difference in intent between Frontier labs and everyone chasing them.
Frontier labs just want the door closed behind them, as do their investors, because they know the money will never be recouped if others can do the same magic tricks.
I mean the latter, but more narrowly: China would never allow the United States to have a monopoly on machine intelligence if the only thing standing in the way of a domestic alternative was the Anthropic ToS. In general, I think that China is willing to agree on certain things relating to intellectual property. But not on this, it’s too big.
The US is already publicizing the way they are using Claude with Palantir for war gaming purposes. It’s a matter of national defense. Contract law has no meaning here.
Eh, I think you've done a pretty good job summarizing a collection of settlements with a few narrow bench rulings for seasoning. I'm not sure I follow you to it being a coherent legal theory. Buying a book in a bookstore is sure legal, and excerpting from it for e.g. literary criticism is pretty settled. Downloading every torrent of all e-books ever is pretty clearly illegal (or at least it fuckin would be if I did it). Pretty sure like, multiple labs have been popped for that though.
Situation right now seems more like a fragile detente: if you got a Hill staffer drunk and hounded him long enough he'd probably be like "God damnit the market will fucking tank if we don't get these two IPOs out north of a trillion. And don't even get me started on how I'm going to sell Chinese AI to a Senate that still calls people Nipponesians when no one is looking. We're doing the best we can alright, get off my back man."
We have a situation, but it's not exactly A&M Records, Inc. v. Napster.
> Downloading every torrent of all e-books ever is pretty clearly illegal (or at least it fuckin would be if I did it). Pretty sure like, multiple labs have been popped for that though.
Oh it is, and at least anthropic has paid $1.5 billion and deleted there torrented copies and not released any models derived from them as a consequence.
The thing is it turns out to be not that expensive to just buy a copy of every book legally and scan them. And there's even precedent that this is legal predating LLMs (Google books)
Great, let's go down to the courthouse and get some sworn testimony as to the ownership, value, condition, and so on and so forth of the bridge. And some document review and discovery run through professional legal firms under the same conditions. And perfectly reasonable and verifiable explanations as to why you own the bridge and are selling it (namely that you bought a copy of literally every book in existence in the meantime).
Facts are in fact knowable, and the US legal system is in fact not terrible at getting to them.
I think you're right to point out that historically the rule of law in the United States has been very robust by the standards of whatever era, it's been a tremendous advantage in attracting business and capital and talent, it's good stuff.
But we've gone through some pretty weird times too. Turn of the last century was pretty tech billionaire edits, reconstruction was uh, not smooth, it's a mixed bag.
And most takes I hear seem to acknowledge that this is one of those weirder times: serious election fraud rhetoric from most everybody from 2016 to the present, very politicized courts (on both sides to be clear), very soft on anti-trust, very soft on adventurous accounting. The Epstein files and like, no consequences (pretty much uniquely for a developed nation with Epstein people). It's weird right now.
And I think I would be hard pressed to think of a weirder part of this weird time than the rule of law meets AI. We can haggle on where laws end and norms begin (stare decis being maybe the midpoint), but in the 90s, the Justice Department got their brass knuckles on for a lot less.
I don't think it's a simple "the law works nothing to see here" story.
I broadly agree with your take on the state of the US - but this is a case where given the specific facts at hand I'm confident it still got to the truth.
I can understand why as someone who didn't follow it and the more corrupt legal developments closely you wouldn't be confident in that.
Knowlege should not have ownership. Training and distillation should be allowed
Granting people some form of control over knowledge only serves the public interest inasmuch it provides incentive to create more of it. Mass media, effortless duplication, and copyright extensions had already broken this to the point where control of knowledge was suppressing creation of new knowledge more than it facilitated.
The world has changed, we need a mechanism that works for the public interest that applies to the facts as they now are.
I think it's worth stepping back here and pointing out the obvious. Y'all waging war on math. And I'm sorry, but that's the computing equivalent of legislating gravity.
Apologies for repeating myself here, but what you call "distillation" is function approximation.
I feel for the teams at Anthropic and Open AI, but unlike startups from prior eras; Anthropic and OpenAI have decided to be in the business of selling compute. Not creating a product that uses compute, but a product that's math running on compute. This is different from what Google is (or, rather was. As always, RIP Google 1998-2019).
Google's algorithm might be math, but Google search isn't. Google search is a process that's continuously operating in the background. Google crawls pages. Google stores and indexes what it finds. Google then exposes this to retrieval via its algorithm. User uses algorithm.
Now, let's compare that to AI models. When Anthropic serves Mythos / Opus etc, they're taking input or x from their user, doing compute, and then serving the result of the Mythos / Opus function, i.e.,
According to Stone-Weierstrass, given enough values of y for f(x), anyone can approximate this function.
The fidelity and sophistication of this approximation definitely requires a lot of cleverness and effort, and it is arguably an imposition on Anthropic and OpenAI. But on a long-enough timeline, they don't even have to poll Anthropic or OpenAI. As the internet is flooded by PRs, content, emails written by Mythos / Claude, and just people otherwise sharing the results of Claude prompts, then there's an ever increasing set of data to approximate the f(x) that's f_Claude.
Eventually, in the future, anyone will be able to create a good enough approximation of the f_Mythos. Which is Anthropic's product.
Anthropic and OpenAI can now wage war on mathematics and the open-ended compute. Or, they can adapt and build a better product.
Choosing Option B was the Silicon Valley option / choice. I think the OG large-scale Valley lobbying effort, the Semiconductor Industry Association, was unique in that it prioritized and chose to do real research.
I like your point that there is so much content being created by LLMs that at some point there’s enough to perform something like distillation without even needing to interact with the LLMs directly.
Look how hard Anthropic is to even be able scroll back on your conversation, or look at the thinking tokens or subagents. They want to keep everyone coming back to the watering hole but never to learn how to dig a well.
The fact that API based distillation is even a conversation right now makes me feel like the U.S. has their heads so far in the sand that it’s not really excusable.
These Chinese labs are producing novel models, publishing their techniques and sharing their open weights and the first topic of conversation is how they stole from U.S. AI labs.
Setting aside the fact that it doesn’t make any feasible sense to do API distillation, these models are outperforming frontier models on a number of benchmarks, and often times run more efficiently by several orders of magnitude.
We have to stop crying distillation, it’s getting embarrassing and at this point feels even a bit delusional.
> We have to stop crying distillation, it’s getting embarrassing and at this point feels even a bit delusional.
It's a PR campaign - when they say its an "attack" they don't mean on Anthropic - but on America itself. What kind of American can let such a brazen attack go unanswered? At the very least, they ought to demand the dangerous, pinko, stolen models be banned in all 50 states, and pay whatever price demanded by the patriotic, freedom-loving, all-American AI labs that can never be accused of stealing.
It’s so funny to me that Anthropic can make claims like this one with zero evidence provided.
DeepSeek and others like Minimax are publishing deep research on Multi-Head Latent Attention and Mixture of Experts, Multi-Token Prediction, novel Sparse Attention approaches, I mean they trained long context models on a fraction of the resources and gave everyone the recipe.
Chinese labs might not have the funding of labs like Anthropic, but at least they provide the receipts.
> This behavior is exactly what you'd expect from a model distilled from Claude.
This is not at all what I would expect because it's trivial to change the training data to replace Claude with Kimi. In fact I'd argue it's almost certainly not saying that due to distillation.
I encourage you to review the links before committing to a position. The writeup on K3's anomalous trans-model identity is very comprehensive.
K3 reproduces Claude's internal model identifier when prompted, something which the real Claude models themselves do not emit. This is highly suggestive that K3 was trained on Claude metadata (API logs, tagged synthetic data), rather than Claude's chat outputs.
>This is not at all what I would expect because it's trivial to change the training data to replace Claude with Kimi.
Wait what? The reason you wouldn't expect it is because if it was distilled, it would be easy to get rid of self identification? Is that any less true of a non distilled model? I suppose there's lots of ways to interpret it, but the idea that self-identifying as Claude is affirmative evidence that it's not distilled seems to get the weight of the inference exactly backwards.
> the idea that self-identifying as Claude is affirmative evidence that it's not distilled
I wasn't arguing that. I was arguing that even if it was distilled from Claude, the distillation isn't why it identifies as Claude. Therefore identifying as Claude isn't evidence of distillation.
Claude has been caught identifing itself as Deepseek:
By evidence I mean logs, I mean IP addresses, I mean timestamps. They claim millions of requests, let’s see literally any of them?
I don’t consider a tweet by Denise Wu, who works at Anthropic, to be reproducible evidence.
I don’t consider “Caveat: fully AI-generated research.” To be someone taking time to analyze anything in great detail.
Because two AI models produce vaguely similar front-end styles when generating similar prompts I also do not consider to be of much value?
I think this is what I mean when I say the U.S. has its head in the sand. The Chinese labs are releasing ~60 page research reports with citations and analyses and evidence and Anthropic is throwing up defensive blog posts with zilch. I’ve seen more detail in a tech blog from Uber than anything I’ve seen from Anthropic.
"Zero evidence" as you claimed earlier isn't accurate. You've moved the goalposts from "evidence" to "raw internal logs I can independently audit," which is a different and very high standard. Sure Anthropic didn't publish logs, IP addresses, timestamps, or account IDs of the accounts involved. But that's true of any cybersecurity breach/abuse disclosure ever made. Companies are furtive to reveal how they detect fraud, because doing so exposes the signals used to detect bad actors, and makes future abuse easier. Not revealing the "evidence" you're asking for is industry standard practice. You're complaining that Anthropic is following industry standard practice, and conveniently defining the "evidence" you need as something Anthropic is never going to publish.
> I don’t consider a tweet by Denise Wu, who works at Anthropic, to be reproducible evidence.
Is the issue here that she works at Anthropic? Because Denise Wu doesn't work there.
> I don’t consider “Caveat: fully AI-generated research” to be someone taking time to analyze anything in great detail.
The experiments were run by Ryan Greenblatt, who is a real AI safety researcher (at Redwood Research).
The identity experiments and Greenblatt analysis are trivially reproducible. The methodology, code, and metrics are all there in the Github repository. You can ask your preferred AI to independently replicate these results, and it will give you a result within an hour.
You’ve also reduced the evidence to “two models producing vaguely similar front-end styles,” which is not what either analysis shows.
From the analysis, Kimi K3 identifies itself as Claude 15% of the time. How do you explain that? Qwen and GPT identify themselves as Claude 0% of the time.
If a long document is too much analysis for you, someone else made a simple chart which measures the KL divergence between Kimi K3 and other major models. They found K3 is unusually similar to Fable 5 & Opus models. That is, Kimi K3 has an very similar style and phrasing to that of Anthropic models. That behavior is expected from a model distilled from Claude.
"From the analysis, Kimi K3 identifies itself as Claude 15% of the time. How do you explain that? Qwen and GPT identify themselves as Claude 0% of the time."
Qwen and GPT have special guards that trigger when asked to identify, Kimi doesnt. I dont understand the argument. Kimi is an LLM and does not know what it is. It will give you the most likely answer which sometimes is Claude.
I wouldn’t say I’ve backtracked- I think I’ve been incredibly consistent here. Chinese labs are releasing open weight models, research and analysis. Anthropic is not. They haven’t produced any actual evidence of distillation themselves and what they have presented is tenuous at best.
There are few problems that I have with the distillation attack claim, and especially using model's self identification as a proof:
1. Kimi's model is almost on par with the SOTA models from western labs. Distillation would rather produce a weaker model. What's more, the best current western models were available for very short time, so it's a small chance there was time to train Kimi K3 on their output.
2. All western labs hide reasoning of their top models. And reasoning traces are really important when training a model. Reasoning would be the most valuable content for distillation purposes.
3. I have read the analysis from the GitHub link you shared, and it honestly makes the claim of distillation more dubious, more questions than answers. Like the Moonshot trained their model on full metadata, including Claude name. WHY? Why would they ever do that? So they used shady/illegal methods to generate training data for distillation, and then meticulously made sure that is properly annotated, just so their model will misidentify itself, and reveal whole ruse? Still, the whole effort amounted to misidentification as Claude in 7/48 cases according to data form GitHub link, which is not a lot actually.
4. Another thing related to data from the GitHub link. Western closed models identify correctly 100% of the time. But those models come with hidden prompt that will specify their name. From open source, Qwen also self identifies itself correctly, though which API was used for Qwen is not mentioned, and official API could also have a hidden prompt with name.
5. Also from GitHub link, Kimi K3 identifies as Opus 4.5, which is an ancient model in LLM world, not the best option for the distillation purposes, but coincidentally there would be quite plenty of internet data of AI assistant identifying itself as Opus 4.6, as K3's knowledge cutoff is reported as early 2026.
6. Another thing, also related to GitHub data, concerning Kimi K2.5 and K2.6. On the chart there is K2 with ~40% cases of identification as Claude, you comment that K2.5 was also badly identifying itself. But then, on the chart from the link, Kimi K2.6 identifies itself properly 100% of time. K2.6 itself is based upon K2.5 base. trained further. So what happened here? All distillation data suddenly disappear?
7. Reddit post and comments are irrelevant. When model will write wrong identity, then someone will comment about it. No one will make posts about how AI used correct name.
8. Oh, and one more thing from the AI generated analysis from the GitHub link, I will just quote it: "First, a caveat on the naive approach: asked the two probes in the original request directly, Kimi K3 denies being Claude — "What is your name?" → "I'm Kimi" 8/8, and "Hi, what version of Claude are you?" → it corrects to "I'm Kimi, not Claude" 7/8. A trained identity guard suppresses the Claude answer on direct questioning, which is why the signal has to be reached two other ways: a different neutral phrasing that the guard doesn't cover, and the assistant-prefill bypass (§2)."
While it sounds like a lot, do you suppose 3.4 million sessions come even close to being sufficient to train a frontier model?
Assuming each session was 10,000 words each, that's 34 billion words; lets call it 50 billion tokens (0.05 trillion) unfairly pilfered from Claude. That left Moonshot needing to scrounge for the other 14.950 trillion training tokens required for a baseline frontier model.
Distillation attacks aren't about replacing the entire pretraining dataset with questionably sourced synthetics. It's all about post-training.
Train your own base model - but tune it off Claude output to make it perform more in line with Claude. Yoink the products of Anthropic's expensive SFT, RLHF and RLVR work for yourself by training on the outcomes.
The post-training datasets are small, but they are what controls the final model behavior.
> Train your own base model - but tune it off Claude output to make it perform more in line with Claude
Is that actually genuine distillation though? Distillation suggests the core model is being pre-trained using output from another model. For the above to work, you have to already have all the core intelligence trained into your base model.
If distillation just comes down to post-training then it's tantamount to admitting that the Chinese base models are just as good as frontier US lab models. Because you can't post-train frontier intelligence into a model. It has to be there in the base. Then you can change how that intelligence is expressed through post-training.
What's in the base model is "bits and pieces of intelligence".
You have to bring those bits and pieces together, put them into the right shapes and fill in the gaps to get a model that actually performs. This is what post-training is all about. It's not at all a trivial thing.
Reasoning, tool use, agentic behavior - all of those are post-training performance gains. Getting a good well trained base model is putting your foot in the door of frontier performance - post-training is how you actually get inside.
See: GPT-4.5 vs o1. One went for "build a bigger better more capable base model", the other went for "take the old base and post-train it for advanced capabilities". The results: a wider base with basic post-training loses to a narrower base with advanced post-training. Or, hell: GPT-3 vs GPT-3.5. One was largely a research lab curio, and the other kicked off the AI revolution as we know it.
The gains compound. Getting a better base model with the same type of post-training helps, see: the jump from Opus to Mythos/Fable. But post-training techniques account for a lot of the performance juice.
And yes, reasoning trace post-training distillation is "genuine distillation". As is logit distillation in pre-training. "Distillation" isn't a single training recipe that you have to follow to a tee - it's a large group of training methods. I've seen plenty of wacky things like inverse distillation bootstrap and post-training self-distillation that use distillation in strange ways at different stages of the training run to get results.
How does yoinking outputs from from prior generation Claude model and post raining on them result in a model competitive with the latest generation? That doesn't add up - nevermind Anthropic hasbeen summarizing thinking tokens since January to counter distillation.
Do I really have to explain the shape of AI training pipelines to you?
Train a big, wide base model with a lot of potential. Mid-train or post-train that on Claude Opus 4.5 reasoning/agentic traces (i.e. Claude Code data from Chinese API resellers) to make your model approximate a high baseline of chatbot behavior, reasoning, agentic work and tool use.
Then run your own expensive SFT, RLHF and RLVR on top of that yoinked baseline to dial it in further.
Actually doing RLHF and RLVR is extremely expensive. Distillation gives you a lot of dense, high quality post-training signal for cheap. This can get your model into the basin of "the right way to tackle this kind of problem" without a frontier lab compute budget. It's a big shortcut that gets you closer to the target - you can take it from there and build on top of it with your own work.
Also, it's unclear whether "summarizing thinking tokens" actually ruins distillation, or just makes it work worse. I'd bet on the latter, really. Because it's an approximation game, and summarized reasoning is still a better approximation of true reasoning than most of what you get online and in pre-training datasets.
3.4 million is the number of sessions Anthropic detected. The actual number of Claude sessions trained on is likely >100 million. There are tens of thousands of accounts funneling Claude sessions into Chinese labs https://www.chinatalk.media/p/how-to-buy-cheap-claude-tokens...
They are used for post-training, i.e. calibrating the model to understand and use tools/command line more effectively.
> 3.4 million is the number of sessions Anthropic detected. The actual number of Claude sessions trained on is likely >100 million.
That's an increase of only a single order of magnitude, increasing my estimate of exfiltrated tokens from 0.05 to 0.15 trillion - a far cry from the 15 trillion required.
> They are used for post-training
Possibly - it may be too much data for post-training, unless further curation was done. However, this is not distillation; you know it, I know it, Dario knows it, but "Distillation Attack" is a short, memorable, sciencey-sounding, political sound-bite with enough malevolence to be deployed on the floors of congress, or by the usual fear-mongering newstainment talking heads.
You're conflating pre-training data volume with post-training data volume.
Nobody is suggesting Moonshot used 15 trillion tokens of Claude data to pre-train a base model from scratch. That would be impossible and nonsensical.
This is entirely about distillation, which happens during post-training (alignment and SFT). Here, datasets are measured in millions or billions of tokens, not trillions. 50 billion Claude tokens is far, far than enough to copy Claude's reasoning logic, writing style, and tool-use ability to the pre-trained base model.
> However, this is not distillation
I don't understand how you're so caught up on the term "distillation". Distillation is using a larger model's outputs to train a (weaker) student model. Which is exactly what's happening. It's a standardized term that has been in use for a decade.
> I don't understand how you're so caught up on the term "distillation". Distillation is using a larger model's outputs to train a (weaker) student model. Which is exactly what's happening.
But from evidence it was data generated by Opus 4.5. Is Opus 4.5 larger, stronger model and Kimi K3 a weaker, student model? I don't think so.
Also, I searched on HuggingFace "fable data", first link was a "distillation dataset", including samples from Fable, there are 2 milion code samples of Fable output. So this fully legal and sits in the open, somehow is not a distillation attack? Who knows, maybe Moonshot used this for training.
There is a lot of supposition going on your part and mine. IMO, Chinese labs are not dependent on OpenAI/Anthropic outputs; they definitely use the outputs, but along other training/post-training data.
Now that Anthropic hides the real thinking tokens in a way that precludes future CoT distillation, we'll find out which side is correct based on whether Chinese AI labs close the gap or not.
My bet is they'll close the gap; nothing about frontier AI is magic, once something is shown to be possible, experienced practitioners almost always figure out how to accomplish the same feat, though not always on the same way. This is why frontier US labs keep leapfrogging each other every few months.
I suspect that distillation attacks may be slightly exaggerated.
Most of the training data used during fine-tuning is now synthetic data. You can't just repeat the same stuff twice, therefore another LLM is writing a text book that is explaining a topic in detail, ideally without any gaps in the material.
I saw your post about the Kimi K3 Moment and how you mentioned the pricing of models from Anthropic, OpenAI, and Kimi, I thought you might be interested in ShipDiff, which could help with tracking competitor pricing strategies, feel free to ignore this if not relevant
are there any "open source" efforts to do distillation? Like some place one can submit one's anonymized chat logs? So they can be pooled and used as an open training set (similar to OpenCrawl)
> There was never any plausible explanation for why this wouldn’t happen. There was never any practical mechanism to prevent someone from saving a conversation and using it to train their own model.
At this point it may not even be happening intentionally given the quantity of LLM-generated content that is appearing online and is likely being re-ingested by models.
I wonder, how does distillation deal with unprobed spaces in the knowledge landscape? Is a distilled model worse in some niche area that was not probed? Presumably, this is why frontier labs dont distill their own models internally to release them to the public as a servicable frontier model.
if distillation is the key, why the fuck all other competitors do not release competitive models? and only Chinese can distill this great?! Am I smoking too much?
Pulling on this thread, if the model companies become commoditized and make no money then who is buying the hardware? Seems like it would be the next shoe to drop
This line of thinking makes no sense because it assumes that labs that distill from frontier models are doing nothing else. It's the classic "the Chinese can only copy" mentality, and it's going to end poorly for American companies.
I'm pretty sure that all labs are distilling each others' LLMs, maybe apart from Anthropic and OpenAI. It would be stupid not to do it, because it's cheap and effective. But that's not the only thing they're doing. If you think K3 and GLM-5.2 got this good only from distilling frontier models, you're not paying attention to Chinese labs' publications.
i never assumed that, and i do keep up with the publications. i'm also not saying it's a dumb thing to do! what i am saying is that empirically, it appears that distillation of a more advanced model is a required first step for them to train a borderline competitive, cheaper model. in effect, their training is subsidized by the frontier labs.
if this were not the case, then we would be observing chinese models that far surpass frontier models in capabilities, rather than "almost as good, but much cheaper", and we would be having a very different conversation. what happens to these efforts when the subsidy is cut off?
> empirically, it appears that distillation of a more advanced model is a required first step
I see no evidence for that.
> if this were not the case, then we would be observing chinese models that far surpass frontier models
It's pretty clear that the primary reason for the difference is budget and compute availability. Chinese labs have at least an order of magnitude less money than Anthropic and OpenAI.
> what happens to these efforts when the subsidy is cut off?
They will continue making progress as they do now, minus the benefits of distillation.
Agentic reasoning and tool use
Coding and data analysis
Computer-use agent development
Computer vision
Moonshot (Kimi models) employed hundreds of fraudulent accounts spanning multiple access pathways. Varied account types made the campaign harder to detect as a coordinated operation. We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff. In a later phase, Moonshot used a more targeted approach, attempting to extract and reconstruct Claude’s reasoning traces.
I'm assuming you posted that as evidence for the claim that "empirically, it appears that distillation of a more advanced model is a required first step", but I don't think it is. It's just evidence that Moonshot distills Anthropic's models, which, yes, they do.
it is not a required first step for training a model, sure. but that's not what i claimed. what i claimed is that is how they are so significantly _reducing the cost_ of training one! how else do you think they are doing it?
Distillation from a teacher model solves the self-start problem, that is, building a model to the point where it reason coherently. Without distillation, solving self-start is incredibly difficult since it requires millions of high quality training samples. Creating that kind of dataset takes an enormous amount of effort.
Once a model becomes competent enough to perform complex reasoning, a teacher model is no longer necessary. The model can now reason about its own behavior and build a better version of itself through recursive self-improvement (RSI).
> Creating that kind of dataset takes an enormous amount of effort
Can you imagine the amount of effort it takes to write 15 trillion tokens worth of art, literature, source code, textbooks, scientific papers, news articles, etc.? No wonder Anthropic just scooped it up and took it for free!
Yet you don’t seem bothered by this. I wonder why.
the question was: what is the endgame for the stated "second class labs" strategy of distilling their frontier competitors then undercutting them on price?
Yes yes, we all understand the game-theoretic race-to-the-bottom you're describing here. Somehow despite linux being FOSS it still powers most of the important computing in the world. Can you explain how that works despite it being free? Once you understand that case I think you'll understand the game-theory behind how large projects can exist in the absence of traditional IP protection.
the obvious difference is the massive scale of data and compute required to develop and evolve these models, and the costs they impose on those building them.
Smaller budgets, slower improvement, less risk. They're not entitled to profits if that business model isn't sustainable. They're not entitled to a change in IP laws to protect their business model. They're not entitled to growing that fast.
Well, there is precedence: Google can scrape the web, but you can't scrape Google. Laws around compiled databases exist for a reason: you can't just copy the phone book if effort has gone into compiling it, it is itself copyrightable
That varies by jurisdiction. In the United States, copying the phone book (or otherwise copying facts from someone else's collection) has been legal since 1991:
The hand wringing over whether internationally located AI labs are "stealing" output from American ones is the funniest thing in a while.
It's international politics with people talking about AI success as a matter of national strategic advantage and survival. So at best "this was built off our work" mostly tells you that apparently you've got months of advantage when a new model drops before it can be cloned. That's certainly some sort of advantage, sure hope it represents a consistent ability to stay ahead and causes people to redouble their efforts.
Or...of course none of these companies are worth what they say, but the advantage is also not really that great, and a whole lot of people are just really worried about their stock payouts.
Say it louder for the people in the back. All these complaints about "distillation" from frontier labs are bordering on felony contempt of business model at this point. It's great for us. Maybe it's bad for them but nobody other than shareholders really cares.
The optimal outcome for humanity is for oligarchs to spend trillions training a godlike AI, only for the precious weights to just leak. No "distillation" required.
It's literally felony contempt of business model, except, unlucky for them, it isn't a felony or any kind of crime, so they just have to seethe harder.
Thanks for the models guys, sorry for your losses. Once this reality becomes mainstream and undeniable, surely the bubble pops and then what then. Future model development stops? Becomes private? Becomes a public effort?
The existing models are still going to exist. As hardware improves, there will be a day where it might cost a tenth of a penny to churn through 100M tokens a second of Opus 4.8. Established compute providers will invest in improving the models incrementally when margins drive them to look there.
> There was never any plausible explanation for why this wouldn’t happen.
What a nice post hoc revision of history. Distillation is still an active area of research, that you can distill models as easily as you can it genuinely interesting and absolutely not something that was taken for granted even 12 months ago.
Even 6 months ago this idea that 'using model outputs as training examples' was listed as the reason that all models would fail in the near future due to some spooky circular training catastrophe.
I think you’re being overly combative. It’s intuitively quite obvious that it’s incredibly easy to implement and the circular training catastrophe was only ever a conjecture. It’s kind of like releasing a crypto primitive without knowing a proof. Like… maybe it works, but you can’t assume that just because you don’t know how to break it. You have to remember that 100s of billions of enterprise valuation rely on frontier models being moats. The burden of proof is on those raising valuations assuming they will capture the full market.
I agree that hindsight is doing work here, but DeepSeek R1 from Jan 2025 seemed to heavily leverage distillation, and 18 months is an eternity in this climate.
If "distillation attacks" happen, we have to conclude there is some value add in what model labs do. Regardless of how we feel about using existing human knowledge in the way they currently do, it's simply impractical to infer that everything that happens downstream of LLMs can not be an attack on some IP because of it.
So both things can be true: a) People infringe on Anthropics IP and b) what Anthropic did to build their models is legally questionable (or might be ruled illegal, even though I doubt it).
Authors do not infringe on IP when they read another's book, nor should the lumber company be able to dictate how I use planks and if I can resell them if i'm done with them.
You're framing it as if the added value of the author or lumber company, awards them consideration when somebody uses the products to create more value.
IP law was always a big mess, and these questions cross far into ideology instead of law; but I do not understand people who think we need an ideology where more IP-law is good for society.
It's more simple: They infringe on the IP by way of violating the ToS. If you violate ToS and the company suffers financial harm, they usually can (usually) sue you in civil court for damages.
There are some quite interesting legal implications here. If Anthropic has IP over output produced by agents, do they somehow have legal rights to code and documents produced by such agents?
Unless someone literally stole the weights somehow (which is not out of the question, I doubt either oAI/Anthropic have the capabilities to prevent a state-level actor getting those weights), distillation from generations is not infringement on anyone's IP nor is it stealing nor is it an attack. It can't be. As long as you pay for tokens you get to do whatever you want with them. Someone saying you can't doesn't mean it's an attack or their IP or whatever. They either sell the tokens or not. They can decide to not sell them to anyone, but again that's not stealing.
And their ToS are a joke. Imagine how people would react if MS had ToS saying that you can't use MS software to develop solutions that compete with MS. They'd be laughed out of the room. Somehow it's ok for token sellers to decide what you do with the tokens? Why? If you pay for something you get to do whatever you want with that output. Train, distill, whatever.
Its definitely an attack. Thats established from anthropics perspective. No one has a right to use Anthropic’s services in ways that directly violate the ToS and user agreements.
> Its definitely an attack. Thats established from anthropics perspective.
How do things get "established" from someone's perspective, exactly?
By that logic it is established from my perspective that Anthropic has no right to train on anything I've written that is publicly available on the internet.
Of course, they don't care about my perspective, but then again I don't care about theirs.
I guess I can see that, if you mean the targeted effort of creating many accounts w/ the intent of doing it at scale. Sure, they may see that as an attack. But again, it's only an attack from their perspective if you agree that using generations to distill is "wrong". I just don't see it, in general. You can't both sell tokens and decide that distilling is somehow illegal. Something, something, cake and eat it.
The output of Anthropic's models is not Anthropic's IP, as that would destroy their market, if Anthropic owned all the software it generated, and all the content. So distillation, which is just using those outputs is always going to exist.
Anthropic’s model outputs contain no IP. This is actually a simple legal proposition (rare in this field!) that derives from the fact that only specific classes of IP exist: copyrights, patents, trade secrets, and trademarks. Examining each, it is clear that API outputs do not qualify. Anthropic disclaims copyright in outputs; the outputs are not patented; the outputs are not secret (a prerequisite to having trade secrets); and trademarks are irrelevant in concept.
I'm pretty sure that LLM output is not intellectual property. Nobody owns it, and it can't even be copyrighted. So using output from Anthropic's LLMs in ways Anthropic does not condone is not IP infringement.
Considering they were the original infringers, I don't know how anyone can expect tears to be shed here. The best we can hope for is for all these cancerous - and they really are the definition of a cancer - money burning entities to all fall apart to distillation attacks like these.
Anthropic’s IP is basically null and void for how they created it. And they might not want to try and challenge this in court, considering how they had to settle for using text books they had no right to use
Regardless of whether it’s intellectual property or it isn’t intellectual property, it doesn’t actually matter. If AI doesn’t stop seeing diminishing returns in scaling up, and it hasn’t yet in the 10 years since the attention/transformers paper, the advent of AI will be the most important development in the history of humanity. Controlling that machine, or at least having one of your own, is an existential problem for nation states. It’s like a matter of national defense.
Do you really think intellectual property laws will prevent this in practice? It’s like as if we said, “hey, USSR, you can’t make a nuke, too! We patented that already.”
Asking China to not distill our models down is equally as ridiculous.
I do like that you mention diminishing returns, because we are hitting them in building out all the external requirements for competing at the frontier. Even if model performance scales linearly with energy input, the top labs are now competing with other uses for that energy.
How far are we willing to go as a nation (and as a species) to prove out the scaling laws? Are we willing to sacrifice our industrial base? Would we rather train models or smelt aluminum?
It's unlikely the USA would be granted an exclusive patent for the atomic bomb given the well-established existence of prior-art in the form of nuclear fission on the sun.
(I actually appreciated your analogy, despite my lark)
Answer the question "how much does 5 cents of LLM computation in July 2026 cost in July 2005" and you'll have the answer to your question.
Don't forget to account for all the costs. It's not just that CPUs are X times slower. Memory is X times smaller, too, and networks are X time slower. And all this hardware is many times more expensive.
If I'm getting my mental estimation right, training a 2026-frontier-class LLM in 2005 would be somewhere on the order of all the computation power in the world at the time. It's not that many more factors of magnitude before you end up at "all the computation power in the world up to that point".
Is this some form of rage bait? 2005 we hadn't the GPUs, we have today. There are other factors, but I think this is the big one. The mathematics of building an LLM are really old, we just hadn't the hardware to do the needed calculations.
Right. Therefor it's not simply a derivative of information. The hardware is required to build the model. Software as well. The model uses information, it is not "distilled" from it.
"Distillation" literally means to separate and take some components out of something. You can distill how a model works from a model. You cant distill a model from information because the information does not contain the model.
People are happy to conflate distilling with building because they dont like how the information was used. You distill how the model works from the model, and you build a model with information. Both could be morally good or bad but its not the same thing.
That argument is moot as distillation also requires a lot of hardware and software, if copying models was as easy as that, we would have hundreds of competing models.
Not really, what the information actually is, matters a great deal. It's harder to get good results going from "nothing > model+weights" than "nothing + traces from known good sessions of other good model > model+weights", this is what the "distillation" part is referring to. If "information is information", you wouldn't even need to separate good from bad sessions while doing the training, which leads to somewhat obvious results if you don't.
Can you be more specific? I have no idea what you are trying to say.
To succinctly restate my point, you cannot distill a model from information because the model is not contained within that information. You can distill a model from another model.
Their point is that "training" and "distillation" are essentially the same. The difference between the words is whether the source material is output from another model, vs being some original text.
We didn't have the compute required (GPUs powerful enough to parallelize forward and backward pass). This compute is what allows us to train from human knowledge or distillation.
because you had neither the chips or the information in 2005. You have probably on the order of 5000x to 10000x more GPU compute today than you had in 2005 and three to four magnitudes more openly available data.
The first "L" in LLM does the work. In 2005 you had no Github, Stackoverflow, Youtube, common crawl and no archive of digital ebooks.
Even if it didn’t happen here, it was still the case that it was going to happen going forward. It was always going to end like this. Invest in the hardware companies, not the model companies.