Log-sum-exp is a real-valued function that takes the natural logarithm of a sum of exponentials. For a vector with finite real components, it is defined by
It is used in convex optimization, statistics, and machine learning both as a smooth approximation to the maximum and as a fundamental operation for calculations in the logarithmic domain. Its derivatives connect it directly to the softmax function, while an equivalent shifted formula allows reliable evaluation even when the individual exponentials would overflow.
Definition and basic properties
The definition uses the exponential function with base . Several identities explain the behavior of log-sum-exp.
Translation equivariance. Adding the same constant to every input adds to the result:
where is the vector whose entries are all one.
Symmetry and monotonicity. Permuting the inputs leaves the result unchanged. Increasing any input, while keeping the others fixed, strictly increases the result.
Bounds by the maximum. If , then
Indeed,
The upper bound is attained when all inputs are equal. For and finite inputs, the lower inequality is strict, although the difference can approach zero when one input dominates the others.
For example,
whereas
is only slightly larger than .
Grouping. Log-sum-exp can be evaluated in stages:
This identity is exact in real arithmetic. Different grouping orders can produce slightly different results in floating-point arithmetic.
Smooth approximation to the maximum
A positive scale parameter , often called a temperature, gives the family
The maximum bounds become
Consequently,
For fixed , the approximation error is bounded by uniformly over all real input vectors. This provides a controlled way to replace a nonsmooth maximum with an infinitely differentiable function.
A related but distinct normalization is
This log-mean-exp expression approximates the maximum from below rather than above. It also approaches the arithmetic mean as , whereas the unnormalized contains the growing term .
Smoothing does not preserve all features of the maximum. At positive temperature, every finite input contributes to the result, and equal maxima receive an additional contribution reflecting their multiplicity.
Derivatives, softmax, and convexity
Define
The gradient of log-sum-exp is the softmax function:
Thus its gradient has positive components summing to one and can be interpreted as a discrete probability distribution over the input indices.
Its Hessian matrix is
For any vector ,
The right-hand side is the variance of the values under probabilities . The Hessian is therefore positive semidefinite, establishing that log-sum-exp is a convex function.
It is not strictly convex on all of : along the all-ones direction, translation equivariance makes it affine. For finite inputs, this is the Hessian’s only null direction. When , its restriction to the subspace is strictly convex.
For the temperature-scaled function,
and
Smaller temperatures concentrate the gradient on the largest inputs. If several inputs tie for the maximum, the gradient converges to equal weights over those maximizing indices as .
Numerically stable evaluation
Directly computing can fail even when the final answer is representable. Large positive inputs may cause exponential overflow; sufficiently negative inputs may cause every exponential to underflow to zero.
The standard stable formula uses :
Every exponential in the shifted sum lies in , and at least one equals . This prevents exponential overflow and prevents the entire sum from underflowing to zero.
For example,
although directly evaluating overflows in common floating-point formats.
For two inputs, a useful form is
The logarithmic correction can be evaluated with log1p, a numerical routine designed to compute accurately when is small. In particular,
is a stable expression for the softplus function.
The shift improves numerical stability but does not make computation exact. Small contributions can still be lost through rounding or underflow, especially in low precision. Such losses are usually negligible for the resulting log-sum, but may matter when individual tiny probabilities or gradients are required. Pairwise or compensated summation can improve accuracy for long sums.
Infinite inputs and empty sums
Implementations must handle special cases separately:
- If at least one input is , the result is .
- If every input is , the result is .
- An empty log-sum-exp is commonly assigned , corresponding to in the extended real numbers.
Applying the shifted formula literally in the first two cases can produce undefined expressions such as or .
Entropy and a variational representation
Let
be the probability simplex. With entropy measured using natural logarithms,
and the convention , log-sum-exp satisfies
The unique maximizing distribution is .
To see this, set . The Kullback–Leibler divergence satisfies
Rearrangement gives
whose maximum occurs at .
More generally,
This characterizes the smoothed maximum as an expected input value augmented by an entropy term. The parameter controls the contribution of entropy.
Applications
Probability calculations in the logarithmic domain
If for nonnegative weights , then
Multiplication of weights becomes addition of their logarithms, while addition of weights becomes log-sum-exp. This permits calculations involving extremely small probabilities without repeatedly converting them back to ordinary scale.
Typical examples include summing mixture components in a Gaussian mixture model, evaluating finite latent-state sums, and implementing the forward recurrence of a hidden Markov model. It also appears when computing marginal likelihoods from sums of weighted likelihood contributions.
Classification losses
For class scores , normalized class probabilities satisfy
If the observed class is , its negative log-probability is
This is the cross-entropy loss for a single categorical target. Implementing the combined expression avoids taking the logarithm of a probability that may already have underflowed to zero.
Convex modeling
For affine expressions , the function
is convex. Such expressions occur in smooth maximum objectives and probabilistic models. Their derivatives are weighted combinations of the affine coefficients, and their Hessians are corresponding covariance matrices.
Weighted variants and limitations
For nonnegative weights , not all zero, a weighted version is
For positive weights, it can be rewritten as
Zero-weight terms may be omitted or represented by an input of .
Negative weights change the problem substantially. The weighted sum may be zero or negative, and cancellation can destroy relative accuracy even when each exponential is computed safely. Signed logarithmic representations therefore require additional handling beyond ordinary log-sum-exp.
The ordinary function also acts as a normalizing operation rather than a guarantee of tractable computation. If a probability model contains exponentially many configurations, rewriting its normalizing sum as log-sum-exp prevents some numerical failures but does not remove the computational cost of evaluating that sum.