{
  "id": "cross-entropy",
  "title": "Cross-entropy",
  "lang": "en",
  "source_lang": "en",
  "summary": "Cross-entropy measures the expected logarithmic loss incurred when one probability distribution is used to describe outcomes generated by another.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "Cross entropy",
    "Cross-entropy loss"
  ],
  "generated_at": "2026-10-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/cross-entropy/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/cross-entropy/",
      "json": "https://aiwiki.page/zh/cross-entropy.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Cross-entropy is a quantity in information theory that measures the average negative logarithm of probabilities assigned by a model distribution to outcomes generated by a reference distribution. It connects probabilistic prediction with information coding: predictions assigning low probability to frequently occurring outcomes incur a larger cost. In machine learning, cross-entropy is widely used as a loss function for fitting probabilistic models and evaluating their predictions. (deeplearningbook.org)"
    },
    {
      "heading": "Mathematical definition",
      "text": "For two discrete probability distributions, \\(P\\) and \\(Q\\), over the same outcome space \\(\\mathcal X\\), cross-entropy is\n\n\\[H(P,Q)=-\\sum_{x\\in\\mathcal X}P(x)\\log Q(x) =\\mathbb E_{X\\sim P}[-\\log Q(X)].\\]\n\nThe expectation is taken under \\(P\\), while the logarithmic probabilities come from \\(Q\\). Thus the order of the arguments matters. Base-two logarithms give units of bits; natural logarithms give nats. Terms with \\(P(x)=0\\) contribute zero by convention. If \\(P(x)>0\\) but \\(Q(x)=0\\), cross-entropy is infinite: the model declares an outcome impossible although the reference distribution allows it. (deeplearningbook.org)\n\nThe quantity \\(-\\log Q(x)\\) is the self-information assigned to outcome \\(x\\) by \\(Q\\). Cross-entropy therefore averages model-assigned surprise rather than the reference distribution’s own surprise. For discrete distributions it is nonnegative, but it need not vanish when the distributions agree. (deeplearningbook.org)"
    },
    {
      "heading": "Relationship to entropy and divergence",
      "text": "Cross-entropy decomposes into Shannon entropy and Kullback–Leibler divergence:\n\n\\[H(P,Q)=H(P)+D_{\\mathrm{KL}}(P\\|Q),\\]\n\nwhere\n\n\\[H(P)=-\\sum_xP(x)\\log P(x),\\qquad D_{\\mathrm{KL}}(P\\|Q)=\\sum_xP(x)\\log\\frac{P(x)}{Q(x)}.\\]\n\nFor finite discrete distributions, Gibbs’ inequality implies \\(H(P,Q)\\geq H(P)\\), with equality precisely when \\(P=Q\\). Consequently, minimizing cross-entropy over \\(Q\\), with \\(P\\) fixed, is equivalent to minimizing this direction of KL divergence. Cross-entropy is not a distance metric: it is generally asymmetric, and \\(H(P,P)=H(P)\\), rather than zero. (cs229.stanford.edu)\n\nIn lossless data compression, ideal code lengths based on \\(Q\\) are \\(-\\log_2Q(x)\\). Their average under the actual source \\(P\\) is cross-entropy, while KL divergence represents the excess over entropy. This interpretation concerns ideal lengths or asymptotic coding rates; individual binary codewords must have integer lengths. (cs229.stanford.edu)"
    },
    {
      "heading": "Statistical estimation and learning",
      "text": "In supervised learning, a model assigns conditional probabilities \\(q_\\theta(y\\mid x)\\) to labels given inputs. For \\(N\\) examples in training data, the empirical objective is\n\n\\[L(\\theta)=-\\frac1N\\sum_{i=1}^{N} \\log q_\\theta(y_i\\mid x_i).\\]\n\nWhen labels are conditionally independent across examples, their likelihood is the product of these probabilities. Taking its logarithm turns the product into a sum, so minimizing \\(L\\) is equivalent to maximum likelihood estimation. The average loss is an empirical estimate of expected predictive logarithmic loss. (deeplearningbook.org)\n\nCross-entropy can serve as the training objective for an artificial neural network. The overall objective may also include a regularization term; in that case it is no longer solely the unmodified negative log-likelihood. Its precise expression depends on the output distribution assumed by the model. (deeplearningbook.org)"
    },
    {
      "heading": "Binary and multiclass forms",
      "text": "For a binary target \\(y\\in\\{0,1\\}\\) and predicted positive-class probability \\(q\\), the Bernoulli cross-entropy is\n\n\\[\\ell(y,q)=-y\\log q-(1-y)\\log(1-q).\\]\n\nThe same formula accepts soft targets \\(y\\in[0,1]\\), interpreted as target probabilities. In multilabel classification, separate binary losses can be calculated for labels that may coexist, rather than forcing all labels into one mutually exclusive distribution. (docs.pytorch.org)\n\nFor \\(K\\) mutually exclusive classes, target probabilities \\(p_k\\) and predictions \\(q_k\\) give\n\n\\[\\ell(p,q)=-\\sum_{k=1}^{K}p_k\\log q_k.\\]\n\nWith one-hot encoding of the correct class \\(c\\), this reduces to \\(-\\log q_c\\). Consequently, only the probability assigned to the correct class appears explicitly, although normalization couples all class probabilities. Implementations may accept a class index instead of an explicit one-hot vector. (docs.pytorch.org)\n\nFor example, assigning probability \\(0.8\\) to the correct class produces approximately \\(0.223\\) nats of loss; assigning \\(0.1\\) produces approximately \\(2.303\\). These values follow directly from the formula and illustrate its strong penalty for confidently incorrect predictions."
    },
    {
      "heading": "Numerical computation",
      "text": "Multiclass models commonly transform raw scores \\(z_k\\) into probabilities using the softmax function:\n\n\\[q_k=\\frac{e^{z_k}}{\\sum_j e^{z_j}}.\\]\n\nFor a normalized target distribution, differentiating cross-entropy with respect to a score gives\n\n\\[\\frac{\\partial\\ell}{\\partial z_k}=q_k-p_k.\\]\n\nThis compact expression supplies an output-layer gradient for backpropagation. It follows algebraically from the softmax and loss formulas. (docs.pytorch.org)\n\nComputing logarithmic probabilities directly from scores avoids unnecessary numerical instability. Multiclass loss can be written using log-sum-exp, while binary implementations can combine the logistic function with the logarithmic loss. Libraries therefore distinguish losses receiving probabilities from those receiving raw scores. Class weights, averaging conventions, and ignored targets also affect the resulting objective. (docs.pytorch.org)"
    },
    {
      "heading": "Language modeling and continuous distributions",
      "text": "A language model assigns conditional probabilities to successive tokens. Average negative log-probability on an independent test set estimates predictive cross-entropy per token. Exponentiating this average gives perplexity, using the same logarithm base. Comparisons require compatible token units and evaluation data. (nlp.stanford.edu)\n\nFor continuous variables with densities \\(p\\) and \\(q\\), the analogous definition is\n\n\\[H(p,q)=-\\int p(x)\\log q(x)\\,dx.\\]\n\nUnlike discrete cross-entropy, this quantity can be negative because a density can exceed one. It relates to differential entropy through the corresponding KL decomposition when the quantities are well-defined; its value depends on the chosen coordinates and reference measure. (deeplearningbook.org)"
    }
  ],
  "links": [
    "information-theory",
    "machine-learning",
    "loss-function",
    "probability-distribution",
    "expected-value",
    "bit",
    "self-information",
    "information-entropy",
    "kullback-leibler-divergence",
    "lossless-data-compression",
    "supervised-learning",
    "training-data",
    "likelihood-function",
    "maximum-likelihood-estimation",
    "artificial-neural-network",
    "regularization",
    "bernoulli-distribution",
    "one-hot-encoding",
    "softmax-function",
    "gradient",
    "backpropagation",
    "logistic-function",
    "language-model",
    "test-set",
    "perplexity",
    "differential-entropy"
  ],
  "sources": [
    {
      "title": "deeplearningbook.org/contents/prob.html",
      "url": "https://www.deeplearningbook.org/contents/prob.html"
    },
    {
      "title": "cs229.stanford.edu/summer2019/ps3.pdf",
      "url": "https://cs229.stanford.edu/summer2019/ps3.pdf"
    },
    {
      "title": "deeplearningbook.org/contents/mlp.html",
      "url": "https://www.deeplearningbook.org/contents/mlp.html"
    },
    {
      "title": "docs.pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html",
      "url": "https://docs.pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html?highlight=criterion"
    },
    {
      "title": "docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html",
      "url": "https://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html"
    },
    {
      "title": "nlp.stanford.edu/fsnlp/mathfound/fsnlp-slides-kl.pdf",
      "url": "https://nlp.stanford.edu/fsnlp/mathfound/fsnlp-slides-kl.pdf"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}