{
  "id": "distilbert",
  "title": "DistilBERT",
  "lang": "en",
  "source_lang": "en",
  "summary": "DistilBERT is a compact Transformer language model trained by knowledge distillation to retain much of BERT’s performance with fewer parameters and faster inference.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "product",
  "aliases": [
    "Distilled BERT"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/distilbert/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/distilbert/",
      "json": "https://aiwiki.page/zh/distilbert.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "DistilBERT is a pretrained language model that compresses BERT through knowledge distillation, transferring information from a larger teacher model to a smaller student model. Introduced in 2019 by Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf at Hugging Face, it was designed to reduce the computational cost of natural language processing while preserving much of BERT’s language-understanding capability. The original English DistilBERT model has approximately 66 million parameters, compared with approximately 110 million in BERT-base, and uses six Transformer encoder layers rather than twelve."
    },
    {
      "heading": "Architecture",
      "text": "DistilBERT retains the encoder-based Transformer architecture used by BERT. Each encoder layer combines multi-head self-attention with a position-wise feed-forward network, supported by residual connections and normalization. Its representations are bidirectional: a token can attend to tokens on both its left and right, subject to the attention mask.\n\nThe original base configuration has:\n\nProperty | DistilBERT configuration\nEncoder layers | 6\nHidden dimension | 768\nAttention heads per layer | 12\nFeed-forward intermediate dimension | 3,072\nMaximum input length | 512 tokens\nParameters | Approximately 66 million\n\nThe principal reduction is in depth, rather than hidden width. Keeping the teacher and student hidden dimensions equal simplifies the transfer of weights and the comparison of their internal representations.\n\nDistilBERT removes BERT’s token-type embeddings, which distinguish sentence segments, and its pooler, which transforms the first token’s final hidden state into a sequence-level representation. It retains token embeddings and learned positional embeddings. Removing token-type embeddings does not prevent it from processing sentence pairs: special separator tokens can still mark boundaries between the two texts.\n\nThe standard English checkpoints use the BERT WordPiece tokenization scheme. The 512-token limit includes special tokens and refers to subword tokens, not necessarily complete words."
    },
    {
      "heading": "Pretraining and knowledge distillation",
      "text": "DistilBERT’s central contribution is to apply distillation during general-purpose pretraining, before adaptation to a particular downstream task. A pretrained BERT teacher supplies training signals to a smaller student, allowing the resulting student to support multiple applications through transfer learning.\n\nThe original training objective combines three components.\n\nDistillation loss\n\nThe student learns to reproduce the teacher’s probability distribution over vocabulary items. Unlike a single target label, this distribution conveys information about alternative predictions and their relative plausibility.\n\nFor vocabulary item \\(i\\), a temperature-scaled distribution can be written as\n\n\\[p_i(T)=\\frac{\\exp(z_i/T)}{\\sum_j \\exp(z_j/T)},\\]\n\nwhere \\(z_i\\) is the model’s logit and \\(T\\) is the temperature. Higher temperatures produce softer distributions. A cross-entropy objective encourages the student’s softened predictions to match the teacher’s.\n\nMasked language modeling\n\nThe student also learns through masked language modeling: selected input tokens are corrupted, and the model predicts their original identities from context. This self-supervised learning objective anchors learning to the training text rather than relying exclusively on the teacher’s predictions.\n\nThe original work used the same broad English text sources as BERT—English Wikipedia and the Toronto BookCorpus—and omitted BERT’s next-sentence prediction objective.\n\nHidden-state alignment\n\nA cosine embedding loss encourages the student’s hidden-state vectors to point in similar directions to the teacher’s corresponding representations. This supplies an additional signal about internal representations, beyond agreement at the output vocabulary.\n\nThe combined loss function can be represented schematically as\n\n\\[\\mathcal{L} = \\alpha \\mathcal{L}_{\\mathrm{distill}} + \\beta \\mathcal{L}_{\\mathrm{MLM}} + \\gamma \\mathcal{L}_{\\mathrm{cos}},\\]\n\nwhere the coefficients control the contributions of the three objectives. The student is initialized using alternating layers from the teacher, rather than entirely random weights. The original experiments found that initialization and the combined training objectives contributed to the resulting performance."
    },
    {
      "heading": "Reported performance",
      "text": "The original paper, DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, reported approximately 40% fewer parameters and approximately 60% faster execution than BERT-base in its evaluation setup, while retaining about 97% of BERT’s performance on the GLUE benchmark.\n\nThese figures describe specific experimental comparisons. The “97%” figure concerns relative aggregate benchmark performance, not a guarantee of retaining 97% of accuracy on every task. Likewise, the speed figure is not a hardware-independent property.\n\nActual inference latency depends on the processor, sequence length, batch size, numerical precision, software implementation, and task-specific prediction head. Parameter count also differs from runtime memory consumption: intermediate activations and attention computations occupy memory in addition to model weights."
    },
    {
      "heading": "Adaptation and applications",
      "text": "DistilBERT generally requires fine-tuning or another task-specific adaptation step to become a useful application model. Common uses include:\n\n- Text classification: sentiment analysis, topic classification, and other document- or sentence-level predictions.\n- Token classification: assigning labels to individual tokens, including named-entity recognition.\n- Extractive question answering: predicting the start and end positions of an answer span in a supplied passage.\n- Sentence-pair classification: determining relationships such as entailment or paraphrase.\n- Information retrieval: serving as a component in specialized retrieval or reranking systems after suitable training.\n\nA standard pretrained DistilBERT checkpoint is not automatically a high-quality sentence-embedding model. Applications that compare texts through vector similarity generally require an appropriate pooling method and training objective.\n\nDistilBERT is an encoder, not an autoregressive conversational model. Its standard masked-token prediction objective does not train it to generate arbitrary continuations in the manner of GPT-style models."
    },
    {
      "heading": "Variants and software availability",
      "text": "Released checkpoints include cased and uncased English models, as well as a multilingual version distilled from multilingual BERT. Task-specific checkpoints add trained prediction heads, so a pretrained base model and a sentiment-analysis or question-answering checkpoint are not interchangeable artifacts.\n\nDistilBERT is distributed through Hugging Face’s Transformers ecosystem, with implementations for frameworks including PyTorch and TensorFlow. The original widely used checkpoints are distributed under the Apache License 2.0."
    },
    {
      "heading": "Limitations and relation to other compression methods",
      "text": "DistilBERT’s smaller capacity can reduce performance relative to its teacher, with the size of the reduction varying by task, training procedure, and evaluation data. Distillation does not guarantee that all teacher capabilities transfer equally well.\n\nIts bidirectional self-attention retains the standard Transformer’s quadratic attention-matrix scaling with sequence length. Reducing the layer count lowers computational cost but does not change that scaling or remove the checkpoint’s input-length limit.\n\nThe student can also inherit statistical associations, biases, and errors from both the teacher and its training data. Compression does not by itself establish robustness, factual reliability, or protection of sensitive information.\n\nDistilBERT differs from quantization, which reduces the numerical precision used to store or compute model parameters. The approaches can be combined: a distilled model can also be quantized. Although its inference cost is lower than BERT-base’s in comparable settings, producing the distilled model still requires access to a teacher and substantial training computation."
    }
  ],
  "links": [
    "language-model",
    "bert",
    "knowledge-distillation",
    "natural-language-processing",
    "transformer-architecture",
    "multi-head-attention",
    "positional-encoding",
    "tokenization",
    "transfer-learning",
    "self-supervised-learning",
    "loss-function",
    "neural-network-inference",
    "fine-tuning",
    "information-retrieval",
    "pytorch",
    "tensorflow",
    "apache-license",
    "training-data",
    "neural-network-quantization"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}