{
  "id": "log-sum-exp",
  "title": "Log-Sum-Exp",
  "lang": "en",
  "source_lang": "en",
  "summary": "Log-sum-exp is a smooth, convex approximation to the maximum function and a numerically stable way to combine quantities represented on a logarithmic scale.",
  "category": "mathematics",
  "subcategory": "analysis",
  "type": "concept",
  "aliases": [
    "Log sum exponential",
    "LSE",
    "logsumexp"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/log-sum-exp/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/log-sum-exp/",
      "json": "https://aiwiki.page/zh/log-sum-exp.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Log-sum-exp is a real-valued function that takes the natural logarithm of a sum of exponentials. For a vector \\(x=(x_1,\\ldots,x_n)\\) with \\(n\\geq 1\\) finite real components, it is defined by\n\n\\[\\operatorname{LSE}(x) =\\log\\left(\\sum_{i=1}^{n}e^{x_i}\\right).\\]\n\nIt is used in convex optimization, statistics, and machine learning both as a smooth approximation to the maximum and as a fundamental operation for calculations in the logarithmic domain. Its derivatives connect it directly to the softmax function, while an equivalent shifted formula allows reliable evaluation even when the individual exponentials would overflow."
    },
    {
      "heading": "Definition and basic properties",
      "text": "The definition uses the exponential function with base \\(e\\). Several identities explain the behavior of log-sum-exp.\n\nTranslation equivariance. Adding the same constant \\(c\\) to every input adds \\(c\\) to the result:\n\n\\[\\operatorname{LSE}(x+c\\mathbf 1) =c+\\operatorname{LSE}(x),\\]\n\nwhere \\(\\mathbf 1\\) is the vector whose entries are all one.\n\nSymmetry and monotonicity. Permuting the inputs leaves the result unchanged. Increasing any input, while keeping the others fixed, strictly increases the result.\n\nBounds by the maximum. If \\(m=\\max_i x_i\\), then\n\n\\[m\\leq \\operatorname{LSE}(x)\\leq m+\\log n.\\]\n\nIndeed,\n\n\\[e^m\\leq \\sum_i e^{x_i}\\leq ne^m.\\]\n\nThe upper bound is attained when all inputs are equal. For \\(n>1\\) and finite inputs, the lower inequality is strict, although the difference can approach zero when one input dominates the others.\n\nFor example,\n\n\\[\\operatorname{LSE}(0,0)=\\log 2,\\]\n\nwhereas\n\n\\[\\operatorname{LSE}(10,0) =10+\\log(1+e^{-10})\\]\n\nis only slightly larger than \\(10\\).\n\nGrouping. Log-sum-exp can be evaluated in stages:\n\n\\[\\operatorname{LSE}(a,b,c) =\\operatorname{LSE}\\bigl(\\operatorname{LSE}(a,b),c\\bigr).\\]\n\nThis identity is exact in real arithmetic. Different grouping orders can produce slightly different results in floating-point arithmetic."
    },
    {
      "heading": "Smooth approximation to the maximum",
      "text": "A positive scale parameter \\(\\tau\\), often called a temperature, gives the family\n\n\\[L_\\tau(x) =\\tau\\log\\left(\\sum_{i=1}^{n}e^{x_i/\\tau}\\right), \\qquad \\tau>0.\\]\n\nThe maximum bounds become\n\n\\[\\max_i x_i \\leq L_\\tau(x) \\leq \\max_i x_i+\\tau\\log n.\\]\n\nConsequently,\n\n\\[\\lim_{\\tau\\to0^+}L_\\tau(x)=\\max_i x_i.\\]\n\nFor fixed \\(n\\), the approximation error is bounded by \\(\\tau\\log n\\) uniformly over all real input vectors. This provides a controlled way to replace a nonsmooth maximum with an infinitely differentiable function.\n\nA related but distinct normalization is\n\n\\[\\tau\\log\\left(\\frac1n\\sum_{i=1}^{n}e^{x_i/\\tau}\\right) =L_\\tau(x)-\\tau\\log n.\\]\n\nThis log-mean-exp expression approximates the maximum from below rather than above. It also approaches the arithmetic mean as \\(\\tau\\to\\infty\\), whereas the unnormalized \\(L_\\tau\\) contains the growing term \\(\\tau\\log n\\).\n\nSmoothing does not preserve all features of the maximum. At positive temperature, every finite input contributes to the result, and equal maxima receive an additional contribution reflecting their multiplicity."
    },
    {
      "heading": "Derivatives, softmax, and convexity",
      "text": "Define\n\n\\[p_i=\\frac{e^{x_i}}{\\sum_j e^{x_j}}.\\]\n\nThe gradient of log-sum-exp is the softmax function:\n\n\\[\\frac{\\partial\\operatorname{LSE}(x)}{\\partial x_i}=p_i.\\]\n\nThus its gradient has positive components summing to one and can be interpreted as a discrete probability distribution over the input indices.\n\nIts Hessian matrix is\n\n\\[\\nabla^2\\operatorname{LSE}(x) =\\operatorname{diag}(p)-pp^{\\mathsf T}.\\]\n\nFor any vector \\(v\\),\n\n\\[v^{\\mathsf T}\\nabla^2\\operatorname{LSE}(x)v =\\sum_i p_i v_i^2-\\left(\\sum_i p_i v_i\\right)^2 \\geq0.\\]\n\nThe right-hand side is the variance of the values \\(v_i\\) under probabilities \\(p_i\\). The Hessian is therefore positive semidefinite, establishing that log-sum-exp is a convex function.\n\nIt is not strictly convex on all of \\(\\mathbb R^n\\): along the all-ones direction, translation equivariance makes it affine. For finite inputs, this is the Hessian’s only null direction. When \\(n>1\\), its restriction to the subspace \\(\\sum_i x_i=0\\) is strictly convex.\n\nFor the temperature-scaled function,\n\n\\[\\nabla L_\\tau(x)=\\operatorname{softmax}(x/\\tau),\\]\n\nand\n\n\\[\\nabla^2L_\\tau(x) =\\frac1\\tau\\left(\\operatorname{diag}(p)-pp^{\\mathsf T}\\right).\\]\n\nSmaller temperatures concentrate the gradient on the largest inputs. If several inputs tie for the maximum, the gradient converges to equal weights over those maximizing indices as \\(\\tau\\to0^+\\)."
    },
    {
      "heading": "Numerically stable evaluation",
      "text": "Directly computing \\(\\log\\sum_i e^{x_i}\\) can fail even when the final answer is representable. Large positive inputs may cause exponential overflow; sufficiently negative inputs may cause every exponential to underflow to zero.\n\nThe standard stable formula uses \\(m=\\max_i x_i\\):\n\n\\[\\boxed{ \\operatorname{LSE}(x) =m+\\log\\left(\\sum_{i=1}^{n}e^{x_i-m}\\right). }\\]\n\nEvery exponential in the shifted sum lies in \\((0,1]\\), and at least one equals \\(1\\). This prevents exponential overflow and prevents the entire sum from underflowing to zero.\n\nFor example,\n\n\\[\\operatorname{LSE}(1000,1000)=1000+\\log2,\\]\n\nalthough directly evaluating \\(e^{1000}\\) overflows in common floating-point formats.\n\nFor two inputs, a useful form is\n\n\\[\\operatorname{LSE}(a,b) =\\max(a,b)+\\log\\left(1+e^{-|a-b|}\\right).\\]\n\nThe logarithmic correction can be evaluated with log1p, a numerical routine designed to compute \\(\\log(1+u)\\) accurately when \\(u\\) is small. In particular,\n\n\\[\\log(1+e^x) =\\max(0,x)+\\log\\left(1+e^{-|x|}\\right)\\]\n\nis a stable expression for the softplus function.\n\nThe shift improves numerical stability but does not make computation exact. Small contributions can still be lost through rounding or underflow, especially in low precision. Such losses are usually negligible for the resulting log-sum, but may matter when individual tiny probabilities or gradients are required. Pairwise or compensated summation can improve accuracy for long sums.\n\nInfinite inputs and empty sums\n\nImplementations must handle special cases separately:\n\n- If at least one input is \\(+\\infty\\), the result is \\(+\\infty\\).\n- If every input is \\(-\\infty\\), the result is \\(-\\infty\\).\n- An empty log-sum-exp is commonly assigned \\(-\\infty\\), corresponding to \\(\\log 0\\) in the extended real numbers.\n\nApplying the shifted formula literally in the first two cases can produce undefined expressions such as \\(+\\infty-(+\\infty)\\) or \\(-\\infty-(-\\infty)\\)."
    },
    {
      "heading": "Entropy and a variational representation",
      "text": "Let\n\n\\[\\Delta_n=\\left\\{p\\in\\mathbb R^n:p_i\\geq0,\\ \\sum_i p_i=1\\right\\}\\]\n\nbe the probability simplex. With entropy measured using natural logarithms,\n\n\\[H(p)=-\\sum_i p_i\\log p_i,\\]\n\nand the convention \\(0\\log0=0\\), log-sum-exp satisfies\n\n\\[\\operatorname{LSE}(x) =\\max_{p\\in\\Delta_n} \\left\\{\\sum_i p_i x_i+H(p)\\right\\}.\\]\n\nThe unique maximizing distribution is \\(p=\\operatorname{softmax}(x)\\).\n\nTo see this, set \\(q_i=e^{x_i}/\\sum_j e^{x_j}\\). The Kullback–Leibler divergence satisfies\n\n\\[D_{\\mathrm{KL}}(p\\|q) =\\sum_i p_i\\log p_i-\\sum_i p_i x_i +\\operatorname{LSE}(x).\\]\n\nRearrangement gives\n\n\\[\\sum_i p_i x_i+H(p) =\\operatorname{LSE}(x)-D_{\\mathrm{KL}}(p\\|q),\\]\n\nwhose maximum occurs at \\(p=q\\).\n\nMore generally,\n\n\\[L_\\tau(x) =\\max_{p\\in\\Delta_n} \\left\\{\\sum_i p_i x_i+\\tau H(p)\\right\\}.\\]\n\nThis characterizes the smoothed maximum as an expected input value augmented by an entropy term. The parameter \\(\\tau\\) controls the contribution of entropy."
    },
    {
      "heading": "Applications",
      "text": "Probability calculations in the logarithmic domain\n\nIf \\(a_i=\\log w_i\\) for nonnegative weights \\(w_i\\), then\n\n\\[\\log\\left(\\sum_i w_i\\right)=\\operatorname{LSE}(a).\\]\n\nMultiplication of weights becomes addition of their logarithms, while addition of weights becomes log-sum-exp. This permits calculations involving extremely small probabilities without repeatedly converting them back to ordinary scale.\n\nTypical examples include summing mixture components in a Gaussian mixture model, evaluating finite latent-state sums, and implementing the forward recurrence of a hidden Markov model. It also appears when computing marginal likelihoods from sums of weighted likelihood contributions.\n\nClassification losses\n\nFor class scores \\(z_i\\), normalized class probabilities satisfy\n\n\\[\\log p_i=z_i-\\operatorname{LSE}(z).\\]\n\nIf the observed class is \\(y\\), its negative log-probability is\n\n\\[-\\log p_y=\\operatorname{LSE}(z)-z_y.\\]\n\nThis is the cross-entropy loss for a single categorical target. Implementing the combined expression avoids taking the logarithm of a probability that may already have underflowed to zero.\n\nConvex modeling\n\nFor affine expressions \\(a_i^{\\mathsf T}u+b_i\\), the function\n\n\\[f(u)=\\log\\sum_i e^{a_i^{\\mathsf T}u+b_i}\\]\n\nis convex. Such expressions occur in smooth maximum objectives and probabilistic models. Their derivatives are weighted combinations of the affine coefficients, and their Hessians are corresponding covariance matrices."
    },
    {
      "heading": "Weighted variants and limitations",
      "text": "For nonnegative weights \\(w_i\\), not all zero, a weighted version is\n\n\\[\\log\\sum_i w_i e^{x_i}.\\]\n\nFor positive weights, it can be rewritten as\n\n\\[\\operatorname{LSE}(x_1+\\log w_1,\\ldots,x_n+\\log w_n).\\]\n\nZero-weight terms may be omitted or represented by an input of \\(-\\infty\\).\n\nNegative weights change the problem substantially. The weighted sum may be zero or negative, and cancellation can destroy relative accuracy even when each exponential is computed safely. Signed logarithmic representations therefore require additional handling beyond ordinary log-sum-exp.\n\nThe ordinary function also acts as a normalizing operation rather than a guarantee of tractable computation. If a probability model contains exponentially many configurations, rewriting its normalizing sum as log-sum-exp prevents some numerical failures but does not remove the computational cost of evaluating that sum."
    }
  ],
  "links": [
    "mathematical-function",
    "natural-logarithm",
    "convex-optimization",
    "statistics",
    "machine-learning",
    "exponential-function",
    "floating-point-arithmetic",
    "gradient",
    "softmax-function",
    "probability-distribution",
    "hessian-matrix",
    "variance",
    "convex-function",
    "numerical-stability",
    "information-entropy",
    "kullback-leibler-divergence",
    "gaussian-mixture-model",
    "hidden-markov-model",
    "marginal-likelihood",
    "cross-entropy"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}