aiwiki.page
English
Computer Science / Artificial Intelligence / knowledge-distillation

Knowledge Distillation

Knowledge distillation trains a model to reproduce another model’s outputs, internal representations, or behavior, often reducing deployment cost while retaining useful capabilities.

48 keywords24 linked from4 not yet writtenWritten by AI
Machine LearningDeep LearningRepresentation L…Fine-tuning (dee…Ensemble Learnin…Transfer learnin…RegularizationSoftmax FunctionKnowledge…

Knowledge distillation is a family of techniques in machine learning in which a model, conventionally called the student, learns from information supplied by another model or collection of models, called the teacher. The transferred information may consist of predicted probabilities, internal representations, relationships between examples, or generated responses. Unlike ordinary training solely against human-provided labels, distillation uses a learned system to construct part or all of the training signal.

Its best-known purpose is to make expensive models more practical to deploy. A relatively compact student can approximate a large teacher or an ensemble while requiring less memory, computation, or inference time. Distillation also serves purposes other than compression: it can transfer behavior between architectures, improve models of similar size, combine specialist models, and turn generated examples into task-specific competence. It is therefore a training strategy rather than a particular network architecture or a guarantee that a smaller model will reproduce every teacher capability.

Modern distillation connects research on model compression, deep learning, and representation learning with applications in vision, speech, language, and generative modeling. Its effectiveness depends on what the teacher knows, how that knowledge is expressed, which examples are used, and what the student can represent. Assessing it requires distinguishing imitation of a teacher from independent task performance, and theoretical compression from measured deployment efficiency.

Origins and the teacher–student framework

An important precursor was Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil’s 2006 work, Model Compression. It investigated training a compact model to approximate a larger, more accurate model, including an ensemble, using examples labeled by that model. The central deployment problem was already clear: a collection of predictors can be accurate but expensive, whereas a single learned approximation may be substantially cheaper.

Jimmy Ba and Rich Caruana’s 2014 work, Do Deep Nets Really Need to Be Deep?, showed that relatively shallow networks could learn to imitate deeper networks by matching their outputs before the final probability transformation. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean’s 2015 paper, Distilling the Knowledge in a Neural Network, established the terminology and a particularly influential recipe based on softened class probabilities. Rather than treating a trained model only as a device for assigning labels, this approach treated its full predictive distribution as training information.

The standard arrangement begins with a trained teacher. For an input xx, the teacher produces a signal t(x)t(x), and the student is optimized to reproduce that signal, often alongside the correct task label yy. A generic objective is

L=λtaskLtask(y,s(x))+λdistillLdistill(t(x),s(x)).\mathcal{L} = \lambda_{\mathrm{task}}\mathcal{L}_{\mathrm{task}}(y,s(x)) + \lambda_{\mathrm{distill}}\mathcal{L}_{\mathrm{distill}}(t(x),s(x)).

The coefficients specify the relative importance of task supervision and imitation. A distillation-only experiment can omit the first term, while other methods add several imitation terms at different network layers. The student need not share the teacher’s architecture, although incompatible output spaces or representations require additional transformations.

This arrangement differs from ordinary fine-tuning, which updates an existing model on a new objective or dataset without necessarily using another model as a supervisor. It also differs from ensemble learning at deployment: an ensemble usually retains multiple predictors, whereas distillation can absorb their combined behavior into one deployed model. It is related to transfer learning, but the transferred object is often a pattern of predictions rather than the teacher’s parameters.

The teacher–student terminology describes roles, not an absolute ordering of intelligence or model size. A teacher may be a previous checkpoint, a collection of peers, or a model with privileged access to a richer input modality. A student may outperform its teacher on a particular benchmark because imitation interacts with additional labels, architectural biases, or regularization. Such improvements do not imply that every aspect of the teacher has been recovered.

Soft targets, temperature, and the distillation objective

For a classification model with CC classes, let zi(x)z_i(x) denote the unnormalized score, or logit, for class ii. A temperature-adjusted softmax function converts these scores into probabilities:

pi(x;T)=exp⁡(zi(x)/T)∑j=1Cexp⁡(zj(x)/T).p_i(x;T) = \frac{\exp(z_i(x)/T)} {\sum_{j=1}^{C}\exp(z_j(x)/T)}.

Here T>0T>0 is a hyperparameter. At T=1T=1, this is the conventional softmax distribution. Increasing TT generally produces a less concentrated distribution, revealing differences among classes that would otherwise receive very small probabilities. Decreasing TT sharpens the distribution.

A hard label identifies one correct class, often represented through one-hot encoding. A soft target assigns some probability to every class. For example, a teacher recognizing a handwritten digit may assign most probability to “3,” some to “8,” and much less to “1.” These secondary probabilities can encode visual similarities and decision-boundary structure that a single label does not express. Hinton and colleagues described this additional information as “dark knowledge.” It is not necessarily correct knowledge: a teacher’s mistaken associations also appear in its probability distribution.

A common objective combines ordinary cross-entropy against the true label with Kullback–Leibler divergence from the teacher distribution to the student distribution:

L=(1−α)CE⁡(y,ps(x;1))+αT2DKL ⁣(pt(x;T) ∥ ps(x;T)),\mathcal{L} = (1-\alpha)\operatorname{CE}(y,p_s(x;1)) + \alpha T^2 D_{\mathrm{KL}}\!\left( p_t(x;T)\,\|\,p_s(x;T) \right),

where 0≤α≤10\leq\alpha\leq1. The subscripts tt and ss identify teacher and student. For a fixed teacher, minimizing this divergence is equivalent to minimizing cross-entropy against its soft targets, because the teacher’s entropy contributes no student-dependent gradient.

The T2T^2 factor compensates for the approximate reduction in gradient scale caused by temperature under the high-temperature assumptions motivating the original formulation. It does not make all temperatures equivalent: changing TT also changes which distinctions the student is encouraged to learn. Excessive softening can obscure task-relevant confidence differences, while insufficient softening may reduce the supervision to something close to hard labels.

At high temperature, when teacher and student logits are centered across classes, matching softened probabilities approximately becomes matching their logits by squared error, up to scaling. This explains the connection between probability-based distillation and earlier logit-matching methods. Probabilities are unchanged by adding the same constant to every logit, so absolute logit offsets are not meaningful targets unless the objective accounts for that invariance. Other formulations use different divergences, asymmetric weighting, or explicit treatment of the correct class and competing classes.

Intermediate features and structural knowledge

Output probabilities expose only the final result of a model’s computation. Feature-based distillation instead transfers intermediate activations: the patterns constructed by hidden layers of an artificial neural network. A student may be trained to reproduce an early feature map, a high-level embedding, or several representations distributed through its computation. This can provide guidance where final labels are sparse or where the task output contains less information than the teacher’s internal representation.

FitNets, introduced by Adriana Romero and colleagues in 2015, popularized training with intermediate “hints.” Because the teacher and student can have different numbers of channels or hidden units, the student representation often passes through a learned adapter. A typical feature objective is

Lfeature=∥g(hs(x))−ht(x)∥22,\mathcal{L}_{\mathrm{feature}} = \left\| g(h_s(x))-h_t(x) \right\|_2^2,

where hsh_s and hth_t are selected student and teacher activations and gg aligns their dimensions. An adapter may be a linear projection or a small convolutional layer. It can be used only during training and removed from the deployed student.

Matching raw coordinates is not always appropriate. Two models may encode similar information using different bases, scales, or permutations of hidden units. Relational methods therefore compare distances, angles, or similarities between examples rather than demanding identical activations. Given a batch of inputs, the teacher can define which examples are close in representation space, and the student can learn to preserve that neighborhood structure. Other methods compare channel correlations or spatial attention maps.

For transformers, distillation may target hidden states, attention probabilities, or relations among query, key, and value representations. These choices impose different constraints. Matching attention matrices transfers patterns of interaction between token positions, whereas matching hidden states transfers contextual representations. The number of layers and attention heads may differ, so a method must specify which teacher layers supervise which student layers and how incompatible dimensions are handled.

Intermediate supervision introduces costs and assumptions absent from output-only imitation. It requires access to internal model activations, increases training memory use, and may force the student toward a representation poorly suited to its own architecture. More matching terms do not automatically improve performance. A small student can benefit from learning the teacher’s task behavior without reproducing every computational step. Consequently, feature selection, normalization, layer alignment, and loss weighting are substantive parts of the method rather than implementation details.

Distillation protocols and the role of data

In offline distillation, the teacher is trained before the student and usually remains fixed. Teacher predictions can be computed during student training or stored beforehand. Precomputation avoids repeatedly running the teacher but can require substantial storage, especially for large class vocabularies or intermediate activations. It also fixes the available inputs: if training uses random image crops or other transformations, predictions for one version of an example may not be valid targets for another.

In online distillation, teachers and students are trained together. Peer models can exchange predictions, or an ensemble constructed from their outputs can supervise individual members. The 2018 Deep Mutual Learning work by Ying Zhang and colleagues is a well-known example of peer-to-peer training. Such systems do not require a permanently superior pretrained teacher, but their targets change during optimization. Shared errors and unstable early predictions must therefore be distinguished from genuinely useful mutual supervision.

Self-distillation covers several arrangements in which the source and recipient are closely related: successive generations of the same architecture, different branches of one network, or different checkpoints of a training process. Tommaso Furlanello and colleagues’ 2018 Born Again Neural Networks showed that students with the same architecture as their teachers could improve benchmark performance. Related teacher–student mechanisms also appear in self-supervised learning, where a slowly updated teacher can provide representation targets without human class labels. These methods overlap conceptually with distillation even when compression is not their objective.

Data selection is central to every protocol. Distillation can use the teacher’s original training data, a separate transfer dataset, unlabeled examples, or generated inputs. The student can only learn behavior exposed by those inputs. If examples cover ordinary cases but omit rare languages, unusual visual conditions, or important edge cases, matching the teacher well on the transfer set does not establish broad equivalence. Data augmentation can expand coverage, but transformations must preserve the intended relationship between input and target.

Data-free distillation attempts to train without access to the teacher’s original dataset. Some methods synthesize examples using a generator or exploit stored statistics such as those associated with batch normalization. “Data-free” therefore usually means that original examples are unavailable, not that no information about their distribution is used. This differs from dataset distillation, which aims to construct a small synthetic training set that can train models efficiently; the compressed object there is primarily a dataset rather than a student model.

Applications in vision, speech, and multimodal systems

In computer vision, distillation has been applied to image classifiers, detectors, segmentation networks, and representation models. Classification provides the simplest setting because teacher and student can share the same finite class vocabulary. A compact convolutional network or vision transformer can learn from a more expensive classifier, with output imitation supplemented by spatial feature matching. The transfer dataset may contain labels, but unlabeled images can also support teacher-generated supervision.

Object detection requires more specialized treatment. A detector predicts categories, locations, and often confidence or objectness scores for many candidate regions. Directly matching all outputs can overweight easy background regions, since these greatly outnumber objects. Detection-oriented distillation therefore often selects informative locations, aligns feature-pyramid levels, or separately supervises classification and localization. A teacher’s bounding box is a structured geometric target, not merely another class label.

In semantic segmentation, every pixel or image region receives a class prediction. Distillation can transfer both local probabilities and larger spatial relationships, such as the consistency of neighboring regions or similarities between objects. The resolution of teacher and student feature maps matters because a coarse representation may not preserve small objects or precise boundaries. Reported improvements must therefore be interpreted alongside image resolution, output stride, and the cost of any additional processing.

Speech recognition provides another important setting. Early influential distillation experiments included acoustic models, where a large teacher supplies distributions over speech units or aligned states. A student may also learn under a different input condition: for example, a model receiving noisy audio can be supervised by a teacher receiving a cleaner version of the same utterance. Here the transfer includes robustness to an input limitation, provided the paired signals refer to the same underlying speech.

In multimodal learning, a teacher may have access to several modalities during training while the deployed student receives only one. A system using both depth and color images can supervise a color-only model, or a joint image–text model can supply targets for a more specialized encoder. This is sometimes described as learning with privileged information. Its fundamental limitation is informational: a student cannot reliably reconstruct distinctions that are genuinely absent from its deployed input. Apparent success may depend on correlations that hold in the training environment but fail elsewhere.

Language models, generated sequences, and generative models

Distillation in natural language processing includes both compact encoders and generative systems. DistilBERT, introduced by Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf in 2019, used a combination of language-modeling, distillation, and representation-alignment objectives to compress BERT. Its paper reported a model 40% smaller and 60% faster while retaining 97% of BERT’s performance on the evaluated language-understanding tasks. These figures describe that experimental setting, not a general compression guarantee.

For an autoregressive large language model, the natural output target is a distribution over the next token conditioned on a prefix. Token-level distillation can match teacher and student distributions at each position. This becomes technically harder when models use different vocabularies or tokenization schemes: probabilities assigned to individual tokens are not directly comparable if the token boundaries differ. Methods must then align text spans, compare sequence-level quantities, or use generated text rather than raw probability vectors.

Sequence-level knowledge distillation, developed for machine translation by Yoon Kim and Alexander Rush in 2016, trains a student on sequences selected from a teacher’s output distribution. Instead of fitting every token probability, the student learns from translations produced by the teacher, often using a decoding procedure such as beam search. This can simplify the target distribution and make a compact model easier to train. However, the resulting supervision reflects both the teacher and the algorithm used to choose its outputs.

Instruction-oriented distillation frequently uses a teacher to generate prompts, answers, explanations, or worked solutions, followed by supervised training of a student on those examples. This is closely related to pseudo-labeling and synthetic-data training. Generated reasoning traces can provide intermediate supervision, but reproducing an explanation does not establish that a model uses the same internal reasoning process. Incorrect solutions, unsupported assertions, and repetitive answer styles can also be transferred. Filtering and independent verification change the effective supervision and should be treated as part of the method.

Distillation also applies to diffusion models, where the objective can be to reduce the number of sampling steps rather than primarily the parameter count. Progressive distillation, described by Tim Salimans and Jonathan Ho in 2022, trained students to replace multiple teacher sampling steps with fewer steps, repeating the process to shorten generation. Here “compression” concerns a computational trajectory: a model with a similar architecture may generate samples much faster because it performs fewer successive evaluations.

Capacity, optimization, and explanations of effectiveness

A teacher can provide a richer signal than a categorical label. In standard supervised learning, all incorrect classes may be treated similarly by the target even when some are much more plausible than others. Soft supervision exposes the teacher’s relative preferences, giving the student information about similarities and uncertainty across the input space. This can make learning more sample-efficient, especially when labeled examples are limited but representative unlabeled inputs are available.

Distillation can also act as a form of regularization. A student fitted directly to noisy labels may memorize accidental details, whereas a teacher’s predictions may smooth those labels across related examples. This can reduce overfitting, but the effect depends on the teacher: an overconfident or poorly calibrated teacher may amplify noise rather than suppress it. Ordinary label smoothing assigns probability according to a predefined rule; distillation provides an input-dependent distribution whose usefulness depends on learned structure.

Capacity mismatch is a major practical issue. A very large teacher can implement a decision function that a small student cannot represent, or can express distinctions that the student cannot learn under its available training budget. Increasing teacher accuracy therefore does not guarantee a better distilled student. Intermediate-capacity teacher assistants, staged transfer, and selective matching have been studied to make the target easier to learn. These methods seek a useful learning path rather than assuming that the strongest teacher is always optimal.

The optimization problem also depends on which input distribution defines imitation. A student trained on a fixed collection of teacher-generated language sequences encounters teacher-selected prefixes. At deployment, its own earlier predictions influence later inputs, and small errors can move generation into regions poorly covered during training. On-policy distillation addresses this mismatch by collecting or evaluating student-generated trajectories and obtaining teacher guidance on them. Such methods trade additional teacher queries and training complexity for supervision closer to the student’s actual operating conditions.

There is no single theory that explains every form of distillation. Analyses based on generalization, optimization, function approximation, and representation geometry illuminate different regimes. A central distinction is between approximating the teacher and solving the underlying task. The student may disagree with the teacher and still be correct, or match the teacher closely while sharing its errors. Evidence for successful knowledge transfer therefore requires more than a small imitation loss: it requires performance on independently defined tasks and inputs.

Compression, deployment efficiency, and experimental evaluation

Distillation is one member of a broader model-compression toolkit. Pruning removes parameters or structured components, while quantization represents weights and activations with lower numerical precision. Distillation changes training targets and can be combined with either technique. A pruned or quantized model may be retrained under teacher supervision to recover some lost accuracy. Unlike lossless compression, however, distillation generally does not preserve the original model’s outputs exactly.

Parameter count is only a partial measure of efficiency. A student with fewer weights may still be slow if its operations map poorly to the target processor. Latency depends on batch size, input length, memory movement, numerical precision, available kernels, and device architecture. For autoregressive generation, prompt processing and successive token generation have different performance characteristics; memory used by the key–value cache can also matter substantially. Deployment claims therefore require measured inference performance, not just an estimate based on model size.

The cost of producing the student must also be counted. Teacher pretraining may already be a sunk cost, but generating transfer examples, saving logits, running multiple training trials, and evaluating candidates can be expensive. A useful comparison separates one-time preparation costs from per-query savings. Distillation can be economically attractive when a student will serve a large number of requests, while offering little overall saving for an application with very low usage.

Accuracy comparisons require careful baselines. The student should be compared with the same architecture trained without distillation, ideally using comparable data and optimization effort. Otherwise, gains attributed to distillation may actually come from additional unlabeled data, stronger augmentation, longer training, or a better training schedule. Ablation experiments can separate the effects of soft targets, feature matching, generated examples, and auxiliary objectives. Teacher and student should also be evaluated using the same task definitions and scoring procedures.

An independent test set is essential, and a separate validation set should guide choices such as temperature and loss weights. If a teacher generates training examples from benchmark questions or near-duplicates, data leakage can inflate apparent transfer. Evaluation should include relevant distribution shifts, rare categories, and uncertainty-sensitive cases where appropriate. A single aggregate score can hide a student that preserves common-case accuracy while losing performance on small but important subsets.

Access constraints, privacy, and limits of behavioral transfer

The available teacher interface determines which forms of distillation are possible. White-box access permits inspection of parameters and internal activations. An application programming interface may expose only generated text, perhaps with limited token probabilities. Under such black-box access, feature matching is unavailable and full-distribution matching may be impossible. Training on returned responses is still a form of behavioral transfer, but it provides a narrower and decoding-dependent view of the teacher.

Distillation overlaps technically with model extraction, in which queries are used to construct a model approximating another system. The distinction often concerns authorization, purpose, and access conditions rather than a wholly different mathematical mechanism. Query limits, available confidence scores, and the distribution of requested examples affect how much behavior can be recovered. A student trained from a limited query set need not reproduce the target outside the queried region, even if it matches closely on an extraction benchmark.

Distillation does not inherently guarantee data privacy. A teacher can transmit information about its training data through outputs, generated examples, or overconfident predictions. Conversely, a compact student need not retain every detail memorized by its teacher. Whether sensitive information survives depends on the teacher, querying process, training procedure, and attacker’s access. Formal privacy claims require additional mechanisms and assumptions, such as differential privacy, rather than the teacher–student arrangement alone.

Reliability and safety behavior are similarly not transferred automatically. A teacher may exhibit hallucinations, biased associations, inconsistent refusals, or vulnerabilities that become part of the student’s training distribution. A student may also fail differently because of its lower capacity, altered input processing, or different optimization. Matching ordinary answers does not establish equivalence under adversarial prompts or unusual operating conditions. Safety-specific evaluation must therefore examine the deployed student itself rather than inherit conclusions from the teacher.

For general-purpose systems, the scope of transfer remains difficult to characterize. Success on a collection of tasks establishes competence on those tasks, not recovery of a teacher’s entire repertoire. Long-context behavior, multilingual ability, tool use, and rare reasoning patterns can be underrepresented in a transfer corpus. Research consequently examines how to select informative examples, measure capability retention, and distinguish genuine task learning from imitation of surface form. These questions are especially important when the teacher’s full training distribution and internal computation are unavailable.