The Student That Beat Its Teacher[WIP].
A plain-language guide to knowledge distillation—and why an AI copy can sometimes outperform the original.
In 2018, researchers reported an experiment that seemed to violate common sense.
They trained a neural network in the usual way. Then they created another network with the same architecture and approximately the same capacity. The second network—the student—did not learn only from the original hard labels. It also learned from the first network’s predictions.
The surprising result was that the student sometimes performed better than its teacher.
These systems became known as Born-Again Neural Networks. Their importance goes beyond an interesting benchmark result. They expose a flaw in the most common explanation of knowledge distillation.
Distillation is usually described as model compression:
Take a large, expensive model, train a smaller model to imitate it, and accept a modest loss in quality in exchange for lower cost.
That description is useful, but incomplete.
A photocopy cannot become sharper than its source. A compressed archive does not contain more information than the original files. And yet a same-sized student can sometimes outperform the model that taught it.
So what is being transferred?
The answer is not simply “the teacher’s answers.” The student receives a more structured learning signal: information about confidence, uncertainty, similarity, decision boundaries and, in some cases, intermediate representations or generated reasoning traces.
The teacher is not merely handing over an answer key.
It is handing over a map.
Born-Again Networks demonstrated that identically parameterized students could outperform their teachers across image-classification and language-modelling experiments, challenging the idea that distillation is useful only for shrinking models.
What the teacher really knows
Imagine that a model is shown a badly handwritten digit.
The correct label says:
This is a 2.
That is all the conventional training label communicates. The correct class receives a one; every other class receives a zero.
But a trained model may see the image differently. It might assign 90% probability to 2, 6% to 7, 3% to 3 and only tiny probabilities to everything else.
Those “wrong” probabilities are not meaningless noise.
They reveal that this particular 2 resembles a 7 more than it resembles a 5. They tell us something about the geometry of the problem: which categories are close, which confusions are natural and where the classification boundary becomes uncertain.
This information is often called dark knowledge. A hard label tells the student what won. A teacher distribution also shows which alternatives nearly won.
That difference matters.
Consider two teachers marking an examination.
The first writes:
Incorrect.
The second writes:
You selected C. That is understandable because C uses the correct principle, but it applies the principle before the required condition is satisfied. B is correct because the condition is checked first.
The second teacher does more than reveal the answer. The explanation exposes the structure of the mistake.
Knowledge distillation does something similar. Instead of training only against hard labels, the student is encouraged to reproduce some part of the teacher’s behaviour, often its probability distribution over possible outputs.
Hinton, Vinyals and Dean formalized this modern form of distillation by training compact models on the softened outputs of larger models or ensembles.
Why temperature is used
The teacher’s less likely predictions may be extremely small. Therefore we apply temperature before calculating the soft probabilities.
A higher temperature does not increase the contrast between probabilities. It does the opposite: it flattens the distribution.
That flattening makes the runner-up possibilities easier for the student to observe.
At normal temperature, the teacher might assign 99.8% to cat, 0.1% to fox and 0.07% to dog. At a higher temperature, the same underlying preference might appear as 72% cat, 15% fox and 9% dog.
The ordering remains informative, but the secondary relationships are no longer buried under tiny numbers.
Core idea: Distillation transfers more than the teacher’s final decision. It transfers selected information about how the teacher distributes belief across possible decisions.
That also explains why a same-sized student can improve. Soft targets can act as a form of regularization, provide richer gradients than one-hot labels and make the learning problem easier to optimize.
The teacher has converted a sparse answer sheet into a structured textbook.
Four decisions that shape what the student learns
Dark knowledge explains the original Born-Again Networks result, but modern distillation transfers more than softened class probabilities. The literature can look like an endless collection of losses, datasets, sampling methods and architectural tricks.
Most of it becomes easier to understand when reduced to four questions:
- What can you observe from the teacher?
- How will you measure disagreement?
- Whose outputs will appear during training?
- Which part of the teacher will the student copy?
Different answers produce different forms of distillation.
Decision 1: Can you look inside the teacher?
The first distinction is between white-box and black-box access.
White-box distillation
With a downloadable or locally deployable model, you may be able to inspect output logits, token probabilities, hidden states, attention activations, embeddings and intermediate representations from individual components.
The student can therefore be trained to imitate more than the teacher’s final text.
It might be asked to match the teacher’s token distribution, reproduce an intermediate representation or align selected hidden layers.
This is closer to studying someone’s rough work than merely copying the final answer.
White-box access does not mean that every part of the teacher is understandable. A billion-dimensional hidden state is not a human-readable explanation. But it gives the distillation system more signals to work with.
Black-box distillation
A model accessed through a conventional API usually exposes far less.
You send a prompt. The system returns text, structured output or another final response. In most cases, you do not receive the complete logit distribution, hidden states or internal activations.
The teacher must therefore be treated as a synthetic-data generator.
A typical pipeline gathers or generates prompts, sends them to the teacher, collects the responses and filters or verifies them before the student is fine-tuned on the resulting dataset.
This is not classic logit distillation in the narrowest sense. It is often better described as response distillation, synthetic-data distillation or imitation through generated demonstrations or whatever new name Chinese labs come up with.
The Self-Instruct family of approaches helped popularize the idea of using a capable language model to produce instruction-following examples at scale.
The difficult part is rarely generation alone. It is quality control.
A synthetic dataset may contain factual errors, repetitive solutions, stylistic artefacts, hidden benchmark contamination, overrepresented task patterns, plausible but invalid reasoning and biases inherited from the teacher.
Strong pipeline therefore uses verifiers, rule-based checks, execution tests, independent models or human review. Asking the teacher to critique its own answer can help, but self-evaluation should not automatically be treated as ground truth.
The quality of a distilled model is bounded not only by the teacher’s ability, but by the quality, coverage and filtering of the teaching data.
Decision 2: What counts as a good imitation?
Suppose you have access to the teacher’s full probability distribution.
You still need to decide what “close to the teacher” means.
For language models, this is not a cosmetic choice. Different disagreement measures can produce students with noticeably different behaviour.
Forward KL: cover what the teacher covers
A conventional distillation objective often minimizes the forward Kullback–Leibler divergence:
In practical terms, the student is penalized when it fails to assign probability to outputs the teacher considers plausible.
This objective tends to be mode-covering. It encourages the student to represent the teacher’s full spread of possibilities.
That sounds desirable, but a small student has limited capacity. Trying to cover every low-probability region of a much larger teacher may force it to spread itself too thin.
Imagine a local band attempting to perform every song recorded by a legendary artist. Its repertoire becomes comprehensive, but the quality of each performance may suffer.
Reverse KL: stay where the teacher is strong
Reverse KL flips the direction:
Now the student is penalized for placing probability in regions where the teacher places little.
This tends to be more mode-seeking. Instead of covering every possibility, the student concentrates on a smaller number of high-probability behaviours.
The band performs fewer songs, but performs them well.
MiniLLM applied reverse KL to generative language-model distillation and reported more precise responses, improved calibration, lower exposure bias and stronger long-form generation than conventional baselines in its evaluated settings.
Neither direction is universally superior.
Forward KL may be useful when coverage matters. Reverse KL may be attractive when precision, commitment and avoidance of low-quality regions matter more.
Other objectives, including Jensen–Shannon divergence, token-level cross-entropy, feature-matching losses and task-specific reward functions—offer additional trade-offs.
The important point is that a distance function is not a neutral mathematical detail.
It determines which mistakes the student is encouraged to avoid.
Decision 3: Whose mistakes does the student practise on?
This is one of the most important and frequently overlooked decisions in language-model distillation.
A simple pipeline collects a fixed dataset of teacher-written responses and trains the student on them. Every training example contains a clean sequence written by the teacher. At deployment, however, the student must continue its own sequence. That creates a mismatch.
During training, the student sees expert-written prefixes. During inference, it sees prefixes containing its own choices—including imperfect ones.
A small mistake early in a response changes the context for every token that follows. The student may then enter a region of the sequence space it never encountered during training. This is a form of exposure bias.
The driving-school analogy is useful.
Watching thousands of hours of professional driving teaches you what a correct journey looks like. It does not teach you how to recover after you take the wrong exit.
On-policy distillation
On-policy distillation addresses this mismatch by allowing the student to generate its own sequences during training.
The teacher then evaluates, scores or corrects the student in the situations the student actually creates.
The student first attempts the task, making its own output part of the training context. The teacher then provides token-level or sequence-level guidance, and the student learns how to improve from states produced by its current policy.
Generalized Knowledge Distillation, or GKD, was designed around this idea. It trains students on self-generated sequences and uses teacher feedback on those sequences, directly targeting the distribution mismatch between training and deployment.
On-policy methods are more computationally demanding because student generations and teacher evaluations must be produced during training.
But they teach a skill that static imitation does not:
recovery from the student’s own mistakes.
Decision 4: What exactly should be copied?
Knowledge can be transferred at several levels.
1. Final answers
The student learns from the teacher’s completed response.
This is the simplest option and works with black-box APIs. It is also the least informative. Many different reasoning processes can produce the same answer, including incorrect processes that happen to end correctly.
2. Token probabilities
The student matches the teacher’s probability distribution over the next token.
This captures uncertainty and relationships between possible continuations, but usually requires access to teacher logits.
3. Intermediate representations
The student is trained to reproduce selected hidden states, attention outputs or feature maps.
This can transfer richer internal structure, although layers of different sizes or architectures may require projection modules to make their representations comparable.
4. Generated reasoning traces
For tasks such as mathematics, coding and planning, the teacher may generate a multi-step rationale before its answer.
The student is then trained on both the solution and the visible working.
This has become especially important in the era of reasoning-focused language models.
DeepSeek’s R1 project released several smaller Qwen- and Llama-based models distilled from R1-generated data. Its report showed that generated reasoning demonstrations could transfer substantial reasoning performance to smaller models.
That result supports a powerful engineering observation:
Reasoning behaviours that are expensive to discover through reinforcement learning may be considerably cheaper to transmit through demonstrations.
But this requires careful wording.
A student that produces convincing step-by-step text has not automatically demonstrated human-like understanding. It may have learned reusable reasoning procedures, superficial patterns or a mixture of both.
The correct question is not:
Does the student look as though it is thinking?
It is:
Does the method improve performance on genuinely new problems, under contamination-controlled evaluation, compared with equally resourced alternatives?
There is also an inference-cost problem.
A model that generates thousands of reasoning tokens for every request may achieve better results while becoming slower and more expensive to operate. The practical objective is therefore not always “copy the longest reasoning trace available.”
Engineers may train on full reasoning traces, retain only high-quality traces or compress them into shorter rationales. Other approaches teach the model when extended reasoning is necessary, use hidden or latent reasoning, or maintain separate fast and deliberate operating modes.
It is also important not to assume that every reasoning model publicly reveals its internal chain of thought. Some systems expose long generated rationales; others reveal only abbreviated reasoning summaries while keeping internal computation hidden.
Visible explanation and internal reasoning are not necessarily the same thing.
This is exactly the reason why you might have observed that while using Claude Fable 5 you cannot see its reasoning/thinking traces.
Why the student can actually win
We can now return to the puzzle in the title.
A same-sized student does not win because it somehow extracts more information from the teacher than the teacher contains. Nor is distillation a lossless copy operation. The student is trained through a different learning process from the one that produced the teacher.
Several effects can make that second process better.
The targets contain more information
The teacher converts a one-hot label into a distribution over alternatives. Instead of receiving one correct class and a wall of zeros, the student receives information about similarity, ambiguity and the shape of the teacher’s decision boundary.
That produces useful gradients even when the teacher and the hard label agree on the winner.
The teacher smooths the learning problem
Real datasets contain ambiguous examples, annotation mistakes and accidental correlations. Hard labels present every example with absolute confidence, even when that confidence is not deserved.
A teacher often responds more softly. Its distribution can act as a learned form of label smoothing: confident where the pattern is clear, uncertain where the example resembles several classes. This discourages the student from fitting every training label as though it were equally reliable.
In that sense, the student may learn a smoother and more generalizable function than the particular function reached by its teacher.
The student gets another optimization attempt
Neural-network training does not identify a unique best set of weights. Initialization, minibatch order, augmentation and optimization noise can lead two identical architectures to different solutions.
The student begins again from a new initialization, but with the teacher’s structure added to the objective. It is not forced to reproduce the teacher’s parameters. It is encouraged to reproduce useful aspects of the teacher’s behaviour while finding its own route through the loss landscape.
Same architecture therefore does not mean same model. It means the student has the same machinery, not the same learned state or training trajectory.
The teacher can compress relationships across examples
The teacher has already integrated patterns from the entire training set into its predictions. Each soft target reflects not only the label attached to one example, but relationships learned across many examples.
The student receives some of that dataset-level structure at every training point. A useful way to think about this is that the teacher has reorganized the original supervision into a curriculum that is easier for the next model to absorb.
These mechanisms are complementary rather than mutually exclusive. Their importance varies by architecture, dataset and distillation objective. There is no theorem saying that a distilled student must improve.
But there is no paradox either.
Core idea: The student is not a photocopy of the teacher. It is a newly trained model learning from a richer, smoother objective built partly from the teacher’s behaviour.
When the student does not win
Born-again improvement is possible, not automatic. A distillation result is convincing only when it survives a few basic checks.
The lesson may be too difficult for the student
A very large teacher can produce distributions or internal representations that a small student cannot approximate. Beyond some capacity gap, a stronger teacher may become a worse instructor.
An intermediate teacher assistant can help translate the signal:
The best teacher is not necessarily the most capable available model. It is the model that provides the most learnable signal for the intended student.
Imitation is not the same as correctness
A student can agree closely with its teacher and still be wrong. It can also disagree and produce the better answer.
Evaluation must therefore measure task performance independently of teacher agreement. It should also check calibration, robustness and the efficiency gain that motivated the student in the first place.
The exam may have leaked into the lesson
When a language model generates training data, it may reproduce benchmark questions, paraphrases or recognizable solution patterns encountered during its own training. A student can then appear to generalize when it has actually studied part of the test.
Exact and semantic deduplication, private evaluations and out-of-distribution tests are essential whenever teacher-generated data is involved.
Specialization can hide a trade-off
A student distilled heavily on code may become better at programming while losing general knowledge, multilingual ability or conversational range. That can be a sensible engineering choice, but it must be reported as a trade-off rather than a universal improvement.
The right claim is rarely “the student is better.” It is usually more specific:
The student is better on these tasks, under these evaluation conditions, at this deployment cost.
Back to the student that beat its teacher
We began with a puzzle: a student with the same architecture as its teacher sometimes achieved better performance.
The result makes sense once distillation is understood as more than copying answers.
A teacher may provide softer targets, relationships between alternatives, calibrated uncertainty, smoother decision boundaries, intermediate representations, worked demonstrations and feedback on mistakes the student generates for itself.
The student then learns under a different objective and training process from the teacher. It may receive a cleaner signal, stronger regularization or a curriculum better suited to its architecture.
That does not mean every distilled student will outperform its teacher.
Most will not.
It means that “teacher” and “student” describe roles in a training process, not a permanent ranking of intelligence.
The result depends on what the student can observe, how disagreement is measured, whose sequences it encounters during training and whether the lesson transfers answers, distributions, representations or reasoning demonstrations. Together, those choices determine whether the student receives a richer lesson or merely a noisier copy of the teacher’s behaviour.
So the next time a small model is described as “punching above its weight,” do not ask only how many parameters it contains.
Ask better questions:
Who taught it?
What information did the teacher expose?
Which mistakes did the student practise recovering from?
What was transferred—and how was success measured?
The size of the student tells you how much machinery it carries.
The teaching process tells you how it learned to use it.
Further reading
- Geoffrey Hinton, Oriol Vinyals and Jeff Dean: Distilling the Knowledge in a Neural Network—the foundational modern distillation paper.
- Tommaso Furlanello and colleagues: Born-Again Neural Networks—the same-architecture student that can outperform its teacher.
- Yuxian Gu and colleagues: MiniLLM—reverse-KL distillation for generative language models.
- Rishabh Agarwal and colleagues: On-Policy Distillation of Language Models—learning from student-generated mistakes through GKD.
- DeepSeek-AI: DeepSeek-R1—reasoning-model training and the release of distilled smaller models.
- Seyed-Iman Mirzadeh and colleagues: Improved Knowledge Distillation via Teacher Assistant—bridging large capacity gaps.