On July 22 the Director of the White House Office of Science and Technology Policy, Michael Kratsios, posted on X that Moonshot AI had distilled Anthropic’s Fable model to build Kimi K3. He disclosed no access logs, no training data indicators, and no procurement documents. The claim rests on Anthropic’s report from February 23, which documented over 16 million Claude exchanges it attributed to distillation campaigns, 3.4 million of them traced to Moonshot.
Yes, we heard this before. In January 2025 it was DeepSeek. The accusation arrives reliably, a few days after a Chinese lab creates a product competitive with the US models, and it is always described as theft.
I want to make a different argument. The panic is not really about distillation, and it is not cope either. It is a fight over who captures the margin on inference, and the technical claims are being selected for effect rather than derived from evidence. Once you look at what distillation actually requires, the Fable story stops making any sense on Anthropic’s published numbers.
Become a paid subscriber to AI Realist and unlock:
full access to your personal AI Realist workspace — all materials, slides, games, and events in one hub. For launch week only: 20% off.
full archive of the articles
direct chat with the response within 48 hours
https://msukhareva.substack.com/subscribe
Distillation in the original sense, from Hinton, Vinyals and Dean in 2015, means training a student to match a teacher’s full probability distribution over every token it considered.
Remember that models predict tokens one by one based on what they learned from the training data. They assign every token a probability score at every step of generation.
The goal of distillation is to reproduce those probabilities in the student model.
The probability prediction is actually in the “logits”, the scores behind the tokens the teacher rejected. A sampled answer is one draw from that distribution. The distribution itself is worth many draws.
After the logits there is a normalisation layer - softmax, that converts raw scores into probability-like scores that scale from 0 to 1.
The most technical and original form of distillation needs the weights or the logprobs. Work presented as SLIM showed that keeping only the top 5 percent of logits per position lets a student reach comparable perplexity with under half the samples that token-level fine-tuning needs.
Qwen used logit-level distillation to produce six of its eight Qwen3 models from two flagships, at roughly one tenth the GPU hours of running the full training pipeline per model.
The rung below is on-policy distillation, where the student generates and the teacher scores every token. This is now standard practice at the frontier. According to a survey of 2026 practice, DeepSeek-V4 trains separate reinforcement learning experts per domain and then distils them into a unified student against reverse KL loss, Nemotron 3 Ultra uses more than ten specialised domain teachers giving token-level guidance, and GLM-5 uses earlier checkpoints as teachers to recover capabilities that sequential reinforcement learning stages degraded.
Simply speaking the student model assigns a logprob to a token at every step of generation and the teacher model also assigns logprobs to the same tokens. KL loss computes how different those scores are.
Obviously, all of this requires logprobs, which proprietary APIs do not give.
Then comes reasoning trace fine-tuning, which needs only visible text. This is what produced the result everyone cites. DeepSeek trained six open models on roughly 800,000 sampled R1 traces using plain supervised fine-tuning, and DeepSeek-R1-Distill-Qwen-32B reached 72.6 on AIME 2024 against 9.3 for GPT-4o. Section 4.2 of the R1 paper reports that running large-scale reinforcement learning directly on the same Qwen2.5-32B base instead scored 47.0. Though the distilled student still fell short of R1 itself at 79.8, so imitation recovered about 91 percent of the teacher on that benchmark.
Simply speaking, it does not need any internal scores, but it needs the reasoning text of every generation.
The bottom rung is using the API as a judge, where the teacher only labels a student's reasoning step good or bad. Here you cannot see how certain the teacher is, so its uncertain guesses arrive looking identical to its confident ones. Measured as a judge, GPT-4o showed an expected calibration error of 39.25 while being correct 49.71 percent of the time. Post-training shapes behaviour strongly, so confidently wrong labels get amplified into hallucinations rather than filtered out.
The appearance of open-weight frontier models means there is no need to overpay for pricey proprietary APIs. AI costs can already be significantly reduced in a compliant and data-safe way.
To do this, join the AI Realist workshop on optimising the AI stack. This is a three-hour, hands-on workshop designed to help participants understand token economics, reduce their AI costs, and achieve better-quality AI results.
The first session, on 30 July, is sold out. The second session will take place on 27 August, and 14 of the 20 places are still available.
Paid AI Realist subscribers receive a 20% discount!
Register here: https://airealist.org/workshops/ai-stack/
Proprietary APIs hide the top rungs. Anthropic offers no logprobs at all, and its current models return a summary of the reasoning rather than the raw chain of thought. The documentation states that no setting returns the raw version, while you are still billed for the full thinking tokens rather than the summary you receive.
The obvious workaround is prompting the model to write its reasoning out in the ordinary answer. That gives you a new output rather than the trace, and real traces need not be readable at all. DeepSeek had to add a language consistency reward to R1 because R1-Zero’s traces drifted out of coherent language, and they published an ablation showing the fix costs performance.
Yet, the generated reasoning is not useless. A June 2026 study measured elicited traces at 0.482 ROUGE-L overlap with a model’s actual internal reasoning, so the traces differ a lot. For distillation purposes, though, elicited traces delivered 96.7 percent of the value of true internal access, 1.25 times the value of summary-based supervision and 2.09 times the value of answer-only supervision. Hiding the trace costs roughly three percent if the attacker can elicit a replacement.
Which is why Anthropic built a classifier to stop the elicitation. Its cookbook page on Fable 5 fallback documents three safety classifier categories, and alongside cyber and bio the third is reasoning_extraction, defined as requests attempting to extract the model’s summarized thinking. If it is triggered, the request reroutes to Opus 4.8, so part of your harvested Fable data would be Opus output. The fallback cannot be disabled at account level, and Anthropic describes the safeguards as deliberately conservative enough that benign technical work sometimes triggers them.
So Anthropic built a purpose-made anti-distillation control, and put it on the exact model the White House says was distilled.
Anthropic published its report on February 23. Fable 5 launched on June 9, was withdrawn on June 12 under US export controls after Amazon flagged a jailbreak, and returned on July 1. Kimi K3 shipped on July 16, with open weights following on July 27. Anthropic's evidence therefore predates Fable by more than three months, and Fable had been reachable for eighteen days when K3 was published.
Three weeks is not fatal on its own. Data collection is fast, and a light post-training pass is cheap. What does not fit is the whole sequence of collecting, filtering, training, evaluating and shipping a 2.8 trillion parameter model with roughly 50 billion active parameters inside that window, and folding Fable data into pretraining would have required a very large cluster to fit at all.
The economics are even worse. On Anthropic’s published pricing Fable 5 costs 10 dollars per million input tokens and 50 per million output, against 3 and 15 for Sonnet 4.6. Its rate limits allow 100,000 output tokens per minute against 400,000 for every other frontier Claude, so a quarter of the extraction throughput or four times as slow. Higher paid tiers do not raise the limits, they only raise the monthly spend cap, which is 500 dollars on the entry tier. That cap is what forces an operation to run thousands of accounts, and it explains the roughly 24,000 fraudulent accounts Anthropic describes far better than evasion does.
Add it up and Fable 5 is the most expensive, slowest and least informative model in Anthropic’s lineup to distil from. It costs 3.3 times more per harvested token than Sonnet 4.6, runs at a quarter of the speed, withholds the raw reasoning while billing you for it, and actively classifies attempts to extract that reasoning.
A serious operation would have used Sonnet 4.6, which launched on February 17, returns the full raw chain of thought, and sits inside the detection window Anthropic reported. Or even better, it would have simply used GLM-5.2 that allows all the complex distillation methods.
It is very hard to see any logical sense in how or why K3 could have been distilled from Fable 5.
Open weights compress margin at the model layer and move value to the infrastructure layer.
Simply speaking it is getting very hard to justify high prices for LLM APIs and the infrastructure becomes much more important.
On February 10 AWS added six fully managed open-weight models to Bedrock: DeepSeek V3.2, MiniMax M2.1, GLM 4.7, GLM 4.7 Flash, Kimi K2.5 and Qwen3 Coder Next. Four are Chinese, two are from labs Anthropic named thirteen days later, and AWS describes them as frontier-class performance at significantly lower inference cost. AWS also resells Anthropic.
Microsoft is currently considering bringing Kimi K3 to Github Copilot.
Cursor fine-tuned the previous version of Kimi and created its own model, Composer.
That is not good news for OpenAI and Anthropic.
Cloud and compute providers like open weight models.
They monetise compute, and an open-weight model they host on their own hardware, in their own region, under their own compliance posture, is a better product for them than reselling somebody else’s API. T
hey keep the customer, they keep the margin, and they can deploy it in environments where sending tokens to a third party is not permitted. Model labs monetise tokens, and every enterprise workload that moves from an API to a self-hosted open model is revenue that does not come back.
Nathan Lambert put the strategic position plainly in his piece on K3 on July 20, arguing that open models massively reduce margin potential for the closed labs and limit their capacity to reinvest. He also reports that the Commerce Department considered adding Chinese labs to its Entity List, that the NSA and National Cyber Director explored advisory warnings, and that officials drafted rules aimed at domestic supply chains using Chinese open-source models. On the substance of the accusation he concludes that if adversarial distillation contributed at all, it did so to a relatively small degree.
Regulatory uncertainty is the product being manufactured here. A CTO does not need to believe that Kimi K3 is stolen. She only needs to suspect that deploying it might become a compliance problem in eighteen months, and the procurement decision defaults back to a proprietary overpriced API. Uncertainty is cheaper to produce than a better model and it works on a much shorter timescale.
And we can see very well that model providers are very concerned and open-weight models are a true danger for their survival. As I am currently writing this, there is a public letter signed by many companies in support of open weights. Even OpenAI signed it, though it might not mean as much.
I looked for the study that would settle this and could not find it. Measuring the effect of API distillation in general is well covered. Microsoft Research distilled from GPT-5-Chat with text-only access and reported gains at 3B, 7B and 14B. NVIDIA’s Minitron work compressed Mistral NeMo 12B into an 8B using 380 billion tokens against the teacher’s 15 trillion, roughly 40 times fewer, and the compressed model beat its teacher on MMLU at 69.5 against 69.0, GSM8K at 58.5 against 56.4 and HumanEval at 36.2 against 23.8.
What nobody has done is attribute a specific released model’s capability to distillation. There is no study that takes an architecture, pretrains it on public data, and then measures what post-training on distilled outputs adds. Only the lab that owns the model can run that counterfactual.
The one paper on detecting distillation in open-weight models is binary. It asks whether a given teacher was used, explicitly offers no way to measure what share of capability came from distillation, and has only been tested on CIFAR-10, ImageNet and diffusion models. It has never been run on a released language model.
Meanwhile the behavioural evidence that does exist establishes contact, not contribution. Zvi Mowshowitz reports that K3 identifies itself as Claude roughly one time in ten when asked, and shows Claude-like formatting habits and sycophancy. He also states that no quantitative analysis attempts to measure distillation’s actual contribution, and the single figure in circulation, Gavin Leech’s estimate of around 15 percent of the outsized gains, is offered as conjecture. His own conclusion is that distillation is clearly nothing close to the whole story.
A joint evaluation by the UK AI Security Institute and the US Center for AI Standards and Innovation found K3 scoring 32.2 percent on ExploitBench against 76.2 for leading US models, and the suggested explanation is that Anthropic’s cyber safety classifiers would strip offensive security content from any Claude-derived corpus. It is a neat story with one problem: GLM-5.2 scored 24.4 percent on the same benchmark, lower than K3, and nobody claims GLM was distilled from Claude. Also, another, more plausible in my opinion, explanation could be that Moonshot simply did not release a powerful cybersecurity model out of safety considerations.
K3’s weights land on July 27, along with a technical report. At that point the architecture becomes independently checkable, and Kimi Delta Attention, Attention Residuals, Stable LatentMoE routing and quantisation-aware training from the supervised fine-tuning stage onward are either real contributions or they are not. No API gives you a novel attention variant. Sampling Claude gives you text.
Distillation is standard practice. Every major lab does it, on its own models, with full access, and publishes the recipe. Calling it an attack is the new part, and it arrived at the same time as the margin compression did. Anthropic is entitled to enforce its terms of service, and thousands of fraudulent accounts is a real security matter that deserves the response it got. That is a separate question from whether distillation explains Kimi K3, and the two are being deliberately run together.
On the specific claim, the numbers do not support it. Fable 5 is the worst target in the lineup on price, on throughput, on what it returns, and on whether it fights back. If a Chinese lab wanted a teacher this summer, the best one available was open, free, unrestricted, and Chinese - GLM-5.2
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.