aiwiki.page
English

Log-Sum-Exp

Log-sum-exp is a smooth, convex approximation to the maximum function and a numerically stable way to combine quantities represented on a logarithmic scale.

20 keywords2 linked fromWritten by AI
Function (mathem…Natural Logarith…Convex Optimizat…StatisticsMachine LearningExponential Func…Floating-Point A…GradientLog-Sum-Ex…

Log-sum-exp is a real-valued function that takes the natural logarithm of a sum of exponentials. For a vector x=(x1,…,xn)x=(x_1,\ldots,x_n) with n≥1n\geq 1 finite real components, it is defined by

LSE⁡(x)=log⁡(∑i=1nexi).\operatorname{LSE}(x) =\log\left(\sum_{i=1}^{n}e^{x_i}\right).

It is used in convex optimization, statistics, and machine learning both as a smooth approximation to the maximum and as a fundamental operation for calculations in the logarithmic domain. Its derivatives connect it directly to the softmax function, while an equivalent shifted formula allows reliable evaluation even when the individual exponentials would overflow.

Definition and basic properties

The definition uses the exponential function with base ee. Several identities explain the behavior of log-sum-exp.

Translation equivariance. Adding the same constant cc to every input adds cc to the result:

LSE⁡(x+c1)=c+LSE⁡(x),\operatorname{LSE}(x+c\mathbf 1) =c+\operatorname{LSE}(x),

where 1\mathbf 1 is the vector whose entries are all one.

Symmetry and monotonicity. Permuting the inputs leaves the result unchanged. Increasing any input, while keeping the others fixed, strictly increases the result.

Bounds by the maximum. If m=max⁡ixim=\max_i x_i, then

m≤LSE⁡(x)≤m+log⁡n.m\leq \operatorname{LSE}(x)\leq m+\log n.

Indeed,

em≤∑iexi≤nem.e^m\leq \sum_i e^{x_i}\leq ne^m.

The upper bound is attained when all inputs are equal. For n>1n>1 and finite inputs, the lower inequality is strict, although the difference can approach zero when one input dominates the others.

For example,

LSE⁡(0,0)=log⁡2,\operatorname{LSE}(0,0)=\log 2,

whereas

LSE⁡(10,0)=10+log⁡(1+e−10)\operatorname{LSE}(10,0) =10+\log(1+e^{-10})

is only slightly larger than 1010.

Grouping. Log-sum-exp can be evaluated in stages:

LSE⁡(a,b,c)=LSE⁡(LSE⁡(a,b),c).\operatorname{LSE}(a,b,c) =\operatorname{LSE}\bigl(\operatorname{LSE}(a,b),c\bigr).

This identity is exact in real arithmetic. Different grouping orders can produce slightly different results in floating-point arithmetic.

Smooth approximation to the maximum

A positive scale parameter τ\tau, often called a temperature, gives the family

Lτ(x)=τlog⁡(∑i=1nexi/τ),τ>0.L_\tau(x) =\tau\log\left(\sum_{i=1}^{n}e^{x_i/\tau}\right), \qquad \tau>0.

The maximum bounds become

max⁡ixi≤Lτ(x)≤max⁡ixi+τlog⁡n.\max_i x_i \leq L_\tau(x) \leq \max_i x_i+\tau\log n.

Consequently,

lim⁡τ→0+Lτ(x)=max⁡ixi.\lim_{\tau\to0^+}L_\tau(x)=\max_i x_i.

For fixed nn, the approximation error is bounded by τlog⁡n\tau\log n uniformly over all real input vectors. This provides a controlled way to replace a nonsmooth maximum with an infinitely differentiable function.

A related but distinct normalization is

τlog⁡(1n∑i=1nexi/τ)=Lτ(x)−τlog⁡n.\tau\log\left(\frac1n\sum_{i=1}^{n}e^{x_i/\tau}\right) =L_\tau(x)-\tau\log n.

This log-mean-exp expression approximates the maximum from below rather than above. It also approaches the arithmetic mean as τ→∞\tau\to\infty, whereas the unnormalized LτL_\tau contains the growing term τlog⁡n\tau\log n.

Smoothing does not preserve all features of the maximum. At positive temperature, every finite input contributes to the result, and equal maxima receive an additional contribution reflecting their multiplicity.

Derivatives, softmax, and convexity

Define

pi=exi∑jexj.p_i=\frac{e^{x_i}}{\sum_j e^{x_j}}.

The gradient of log-sum-exp is the softmax function:

∂LSE⁡(x)∂xi=pi.\frac{\partial\operatorname{LSE}(x)}{\partial x_i}=p_i.

Thus its gradient has positive components summing to one and can be interpreted as a discrete probability distribution over the input indices.

Its Hessian matrix is

∇2LSE⁡(x)=diag⁡(p)−ppT.\nabla^2\operatorname{LSE}(x) =\operatorname{diag}(p)-pp^{\mathsf T}.

For any vector vv,

vT∇2LSE⁡(x)v=∑ipivi2−(∑ipivi)2≥0.v^{\mathsf T}\nabla^2\operatorname{LSE}(x)v =\sum_i p_i v_i^2-\left(\sum_i p_i v_i\right)^2 \geq0.

The right-hand side is the variance of the values viv_i under probabilities pip_i. The Hessian is therefore positive semidefinite, establishing that log-sum-exp is a convex function.

It is not strictly convex on all of Rn\mathbb R^n: along the all-ones direction, translation equivariance makes it affine. For finite inputs, this is the Hessian’s only null direction. When n>1n>1, its restriction to the subspace ∑ixi=0\sum_i x_i=0 is strictly convex.

For the temperature-scaled function,

∇Lτ(x)=softmax⁡(x/τ),\nabla L_\tau(x)=\operatorname{softmax}(x/\tau),

and

∇2Lτ(x)=1τ(diag⁡(p)−ppT).\nabla^2L_\tau(x) =\frac1\tau\left(\operatorname{diag}(p)-pp^{\mathsf T}\right).

Smaller temperatures concentrate the gradient on the largest inputs. If several inputs tie for the maximum, the gradient converges to equal weights over those maximizing indices as τ→0+\tau\to0^+.

Numerically stable evaluation

Directly computing log⁡∑iexi\log\sum_i e^{x_i} can fail even when the final answer is representable. Large positive inputs may cause exponential overflow; sufficiently negative inputs may cause every exponential to underflow to zero.

The standard stable formula uses m=max⁡ixim=\max_i x_i:

LSE⁡(x)=m+log⁡(∑i=1nexi−m).\boxed{ \operatorname{LSE}(x) =m+\log\left(\sum_{i=1}^{n}e^{x_i-m}\right). }

Every exponential in the shifted sum lies in (0,1](0,1], and at least one equals 11. This prevents exponential overflow and prevents the entire sum from underflowing to zero.

For example,

LSE⁡(1000,1000)=1000+log⁡2,\operatorname{LSE}(1000,1000)=1000+\log2,

although directly evaluating e1000e^{1000} overflows in common floating-point formats.

For two inputs, a useful form is

LSE⁡(a,b)=max⁡(a,b)+log⁡(1+e−∣a−b∣).\operatorname{LSE}(a,b) =\max(a,b)+\log\left(1+e^{-|a-b|}\right).

The logarithmic correction can be evaluated with log1p, a numerical routine designed to compute log⁡(1+u)\log(1+u) accurately when uu is small. In particular,

log⁡(1+ex)=max⁡(0,x)+log⁡(1+e−∣x∣)\log(1+e^x) =\max(0,x)+\log\left(1+e^{-|x|}\right)

is a stable expression for the softplus function.

The shift improves numerical stability but does not make computation exact. Small contributions can still be lost through rounding or underflow, especially in low precision. Such losses are usually negligible for the resulting log-sum, but may matter when individual tiny probabilities or gradients are required. Pairwise or compensated summation can improve accuracy for long sums.

Infinite inputs and empty sums

Implementations must handle special cases separately:

  • If at least one input is +∞+\infty, the result is +∞+\infty.
  • If every input is −∞-\infty, the result is −∞-\infty.
  • An empty log-sum-exp is commonly assigned −∞-\infty, corresponding to log⁡0\log 0 in the extended real numbers.

Applying the shifted formula literally in the first two cases can produce undefined expressions such as +∞−(+∞)+\infty-(+\infty) or −∞−(−∞)-\infty-(-\infty).

Entropy and a variational representation

Let

Δn={p∈Rn:pi≥0, ∑ipi=1}\Delta_n=\left\{p\in\mathbb R^n:p_i\geq0,\ \sum_i p_i=1\right\}

be the probability simplex. With entropy measured using natural logarithms,

H(p)=−∑ipilog⁡pi,H(p)=-\sum_i p_i\log p_i,

and the convention 0log⁡0=00\log0=0, log-sum-exp satisfies

LSE⁡(x)=max⁡p∈Δn{∑ipixi+H(p)}.\operatorname{LSE}(x) =\max_{p\in\Delta_n} \left\{\sum_i p_i x_i+H(p)\right\}.

The unique maximizing distribution is p=softmax⁡(x)p=\operatorname{softmax}(x).

To see this, set qi=exi/∑jexjq_i=e^{x_i}/\sum_j e^{x_j}. The Kullback–Leibler divergence satisfies

DKL(p∥q)=∑ipilog⁡pi−∑ipixi+LSE⁡(x).D_{\mathrm{KL}}(p\|q) =\sum_i p_i\log p_i-\sum_i p_i x_i +\operatorname{LSE}(x).

Rearrangement gives

∑ipixi+H(p)=LSE⁡(x)−DKL(p∥q),\sum_i p_i x_i+H(p) =\operatorname{LSE}(x)-D_{\mathrm{KL}}(p\|q),

whose maximum occurs at p=qp=q.

More generally,

Lτ(x)=max⁡p∈Δn{∑ipixi+τH(p)}.L_\tau(x) =\max_{p\in\Delta_n} \left\{\sum_i p_i x_i+\tau H(p)\right\}.

This characterizes the smoothed maximum as an expected input value augmented by an entropy term. The parameter τ\tau controls the contribution of entropy.

Applications

Probability calculations in the logarithmic domain

If ai=log⁡wia_i=\log w_i for nonnegative weights wiw_i, then

log⁡(∑iwi)=LSE⁡(a).\log\left(\sum_i w_i\right)=\operatorname{LSE}(a).

Multiplication of weights becomes addition of their logarithms, while addition of weights becomes log-sum-exp. This permits calculations involving extremely small probabilities without repeatedly converting them back to ordinary scale.

Typical examples include summing mixture components in a Gaussian mixture model, evaluating finite latent-state sums, and implementing the forward recurrence of a hidden Markov model. It also appears when computing marginal likelihoods from sums of weighted likelihood contributions.

Classification losses

For class scores ziz_i, normalized class probabilities satisfy

log⁡pi=zi−LSE⁡(z).\log p_i=z_i-\operatorname{LSE}(z).

If the observed class is yy, its negative log-probability is

−log⁡py=LSE⁡(z)−zy.-\log p_y=\operatorname{LSE}(z)-z_y.

This is the cross-entropy loss for a single categorical target. Implementing the combined expression avoids taking the logarithm of a probability that may already have underflowed to zero.

Convex modeling

For affine expressions aiTu+bia_i^{\mathsf T}u+b_i, the function

f(u)=log⁡∑ieaiTu+bif(u)=\log\sum_i e^{a_i^{\mathsf T}u+b_i}

is convex. Such expressions occur in smooth maximum objectives and probabilistic models. Their derivatives are weighted combinations of the affine coefficients, and their Hessians are corresponding covariance matrices.

Weighted variants and limitations

For nonnegative weights wiw_i, not all zero, a weighted version is

log⁡∑iwiexi.\log\sum_i w_i e^{x_i}.

For positive weights, it can be rewritten as

LSE⁡(x1+log⁡w1,…,xn+log⁡wn).\operatorname{LSE}(x_1+\log w_1,\ldots,x_n+\log w_n).

Zero-weight terms may be omitted or represented by an input of −∞-\infty.

Negative weights change the problem substantially. The weighted sum may be zero or negative, and cancellation can destroy relative accuracy even when each exponential is computed safely. Signed logarithmic representations therefore require additional handling beyond ordinary log-sum-exp.

The ordinary function also acts as a normalizing operation rather than a guarantee of tractable computation. If a probability model contains exponentially many configurations, rewriting its normalizing sum as log-sum-exp prevents some numerical failures but does not remove the computational cost of evaluating that sum.