The maximum entropy principle is a method for assigning a probability distribution when available information does not determine one uniquely. It selects the distribution with the largest entropy among all distributions satisfying the stated constraints. These constraints may specify possible outcomes, mean values, correlations, or other known properties. The principle therefore makes a precise distinction between information supplied to a model and additional structure that the model would otherwise introduce.
Maximum entropy connects information theory, statistics, and statistical mechanics. It explains why familiar distributions arise from particular combinations of support and moment constraints, leads naturally to exponential-family models, and provides methods for reconstructing distributions from incomplete observations. Its interpretation requires care: the result depends on the chosen state space, constraints, and reference measure, and maximizing entropy is not by itself a guarantee of empirical accuracy.
Mathematical formulation and meaning
For a finite collection of outcomes , write for the probability assigned to . Using natural logarithms, Shannon entropy is
with . A change of logarithm base changes only the unit of entropy, not its maximizer. A typical maximum entropy problem specifies the expected values of functions :
The task is to maximize subject to these equations, , and . Support restrictions, such as ruling out physically impossible outcomes, are also part of the information supplied to the problem.
With normalization as the only constraint, the answer is the uniform distribution, . This follows from
with equality only for equal probabilities. If some outcomes are forbidden, uniformity applies only to the remaining allowed outcomes. Thus “assign equal probabilities in the absence of information” presupposes a definite list of alternatives; it does not specify how that list should be constructed.
The optimization has particularly strong properties on a finite state space. Entropy is strictly concave, and linear expectation constraints define a convex set. Whenever this set is nonempty, it is closed and bounded, so an entropy maximizer exists. Strict concavity makes that maximizer unique, even if some coordinates are forced to zero. The mathematical problem can therefore have a unique answer although many distributions initially satisfy the constraints.
Calling this answer “least informative” has a specific meaning. It has the greatest Shannon entropy within the chosen feasible class, not the least information according to every conceivable measure. Nor does it mean that its probabilities must coincide with unknown physical frequencies. It is an assignment conditional on the model’s stated information.
For example, knowing the mean outcome of a six-sided die does not generally determine its six face probabilities. Maximum entropy selects one distribution from the remaining possibilities. Adding a variance constraint changes the feasible class and usually changes the answer. The principle does not determine which measurements are trustworthy, whether the die has six possible outcomes, or whether the constraints describe the phenomenon adequately; these are separate modeling questions.
Historical development
The principle grew from nineteenth-century work on the relationship between microscopic states and macroscopic properties. Ludwig Boltzmann’s 1877 statistical treatment of entropy connected a macrostate’s entropy with the number of microscopic arrangements compatible with it. The relation commonly written expresses the idea that macrostates corresponding to more microscopic arrangements have greater entropy. This counting argument was an important precursor, though it is not identical to maximizing Shannon entropy over arbitrary probability assignments.
J. Willard Gibbs developed ensemble methods systematically in his 1902 book Elementary Principles in Statistical Mechanics. Rather than following a single microscopic trajectory, Gibbs represented a system through a probability distribution over possible states. His canonical ensemble assigns exponentially decreasing probabilities to states of increasing energy when a system is in thermal equilibrium with a reservoir. The associated entropy functional has the same mathematical form, apart from a physical constant, as the discrete information entropy introduced later.
In 1948, Claude Shannon established entropy as a central quantity in communication theory. There, entropy measures the average information associated with a source’s uncertain output and determines fundamental limits on lossless coding. Shannon’s mathematical formulation made entropy useful independently of any specifically thermodynamic interpretation. Its properties, including additivity for independent sources and concavity under mixing, provided a framework for comparing uncertain distributions.
Edwin Thompson Jaynes explicitly formulated maximum entropy as a general method of statistical inference in his two 1957 papers titled “Information Theory and Statistical Mechanics.” He argued that equilibrium distributions could be derived by maximizing information entropy subject to known macroscopic expectations. In this interpretation, a statistical ensemble represents the information available about a system rather than necessarily describing a literal collection of replicas.
Later developments clarified both the method’s scope and its assumptions. Solomon Kullback’s work on information measures supported minimum-relative-entropy formulations that update a nonuniform reference distribution. Imre Csiszár developed information-projection methods and their geometry. John Shore and Rodney Johnson’s 1980 axiomatic treatment examined consistency requirements for inference under constraints.
During the late twentieth century, maximum entropy also became a practical modeling technique in spectral estimation, image reconstruction, language processing, and ecological prediction. These developments did not establish a single universal interpretation of probability. Instead, the same optimization structure acquired several readings: an inference rule, an equilibrium characterization, a constrained estimator, and a computational device for selecting among underdetermined models.
Variational derivation and exponential families
The characteristic exponential form follows from constrained optimization. Suppose the maximizing distribution is positive on the allowed finite state space. Introducing Lagrange multipliers for normalization and moment constraints gives
Setting each partial derivative with respect to equal to zero yields
Consequently,
The normalizing quantity is the partition function, and these distributions form an exponential family.
Define the log-partition function . Its first derivatives are model expectations:
The constraints are therefore satisfied by choosing so that . Its second derivatives give the covariance matrix of the constrained features:
This matrix is positive semidefinite, establishing the convexity of . It is positive definite when no nontrivial linear combination of the features is constant over the support.
These identities connect the original entropy problem to Lagrangian duality. Under suitable feasibility conditions, the dual problem is
At an interior optimum, the maximum entropy equals . The dual gradient is the discrepancy between predicted and required feature means, giving a direct computational interpretation of optimization steps.
The exponential formula needs qualification at the boundary. If a constraint requires an event to have probability zero, no finite exponential parameter can produce that zero while retaining a positive base measure there. The entropy maximizer still exists on a finite feasible state space, but it may belong only to the closure of the exponential family. One can then reduce the support or consider diverging parameter sequences.
Distributional uniqueness also differs from parameter uniqueness. If two features differ only by a constant, their multipliers are redundant even though the resulting probability distribution is uniquely determined. Removing redundant constraints improves identifiability and numerical conditioning. More generally, attainable expectation vectors lie in the convex hull of the feature vectors ; their position within that hull determines whether ordinary finite multipliers can represent the solution.
Distributions selected by common constraints
Many familiar distributions have maximum entropy characterizations, but each characterization includes its support and reference measure. On a finite set, normalization alone produces equal probabilities. On a fixed interval , maximizing differential entropy relative to ordinary length measure gives the constant density , with entropy . Neither statement extends to an unrestricted uniform distribution on the entire real line, because no such probability distribution exists.
For a nonnegative real-valued variable with fixed mean , the maximizer is the exponential distribution:
Its entropy is . The discrete counterpart, on the nonnegative integers with fixed mean , is geometric:
The difference between these answers illustrates that support is substantive information, not a technical afterthought.
On the real line, fixed mean and variance select the normal distribution:
For any density with those moments and well-defined entropy,
with equality only for the Gaussian density, up to equality almost everywhere. In dimensions, fixed mean vector and positive-definite covariance matrix select the multivariate normal distribution, with entropy
Different constraints produce different shapes. A fixed value of on the real line selects a density proportional to , the Laplace form. On the positive real line, simultaneous constraints on and , when feasible with suitable parameters, give the gamma distribution. These results arise because the constrained functions become terms in the exponential.
Care is needed with plausible but incorrect characterizations. A Poisson distribution does not maximize Shannon entropy over all distributions on nonnegative integers with a given mean; the geometric distribution does. A Poisson law can instead arise from a relative-entropy problem with weights proportional to , or from additional stochastic assumptions. Similarly, a mean constraint alone on the whole real line does not select a normal distribution: distributions centered at that mean can have arbitrarily large variance and entropy.
Continuous variables and relative entropy
For a continuous random variable with density relative to a measure , the direct analogue of discrete entropy is
With Lebesgue measure this is differential entropy. Unlike discrete entropy, it can be negative, and its numerical value depends on the coordinate system. If is a smooth invertible transformation, then, under suitable integrability conditions,
A nonlinear coordinate change can therefore alter which density maximizes bare differential entropy unless the reference measure is transformed consistently.
This dependence is substantive. A distribution uniform in a positive length is not uniform in its logarithm . Equal intervals in those coordinates represent different partitions of possibilities. Maximum entropy cannot decide between them without additional information about what measure represents the baseline weighting of states. In physical applications that measure may come from phase-space geometry or microscopic counting; in other applications it is part of the statistical model.
A general formulation uses a specified reference probability distribution and minimizes Kullback–Leibler divergence:
The selected distribution is
where expresses the constraints. Because the density ratio transforms consistently, relative entropy is invariant under invertible coordinate transformations applied to both distributions. For a uniform on a finite set, minimizing this divergence is exactly equivalent to maximizing Shannon entropy.
Under expectation constraints, the resulting density is
where normalizes the distribution. A nonuniform reference therefore remains visible in the answer. If assigns zero probability to an event, assigning that event positive probability gives infinite divergence; finite-divergence updating cannot introduce support absent from the reference.
On infinite spaces, existence is no longer automatic. The partition function must be finite, the requested moments must be attainable, and the optimizing sequence must not lose mass toward infinity. Some constraints admit no maximum, even when increasingly high entropy values are possible. Results about existence thus require assumptions on support, moments, tightness, or integrability. Formal use of multipliers alone does not establish that a normalized maximizing distribution exists.
Statistical estimation and information projection
Maximum entropy has a direct relationship to maximum likelihood estimation. Let observations supply empirical feature averages
For the model
the log likelihood, apart from a term independent of , is
Maximizing it is therefore equivalent to solving the dual of the minimum-relative-entropy problem whose feature expectations equal . The same fitted distribution can be described either as a constrained entropy solution or as a likelihood fit within an exponential family.
This equivalence does not make entropy maximization and likelihood maximization identical procedures in every setting. They optimize over different objects: the former usually varies a distribution subject to informational constraints, whereas the latter varies parameters within a specified sampling model. Their equivalence depends on matching the feature functions, reference measure, and empirical constraints.
The geometric interpretation is an information projection. For finite state spaces and linear equality constraints, suppose the minimizing distribution is positive on the relevant support. Every other feasible then satisfies
The identity follows because is an affine combination of the constrained features, whose expectations agree under and . It explains in what sense is closest to the reference. For broader convex constraint sets, an appropriate inequality generally replaces the exact identity.
Maximum entropy is also related to, but distinct from, Bayesian inference. Bayesian conditioning combines a prior distribution with a likelihood function. A relative-entropy update instead selects the smallest divergence change satisfying new constraints. Ordinary conditioning on an event of positive reference probability is a special case: minimizing divergence over distributions supported on that event gives the reference distribution conditioned on it.
When constraints are estimated from finite samples, exact moment matching can reproduce sampling noise. Regularization may penalize large multipliers, allow moment discrepancies, or replace equalities with uncertainty intervals. A full Bayesian treatment may instead assign distributions to parameters and propagate their uncertainty. The unregularized maximum entropy solution supplies neither confidence intervals nor a posterior distribution automatically. Such uncertainty assessments require a sampling model or another explicit account of how the constraints were obtained.
Equilibrium physics and distributions over trajectories
In equilibrium thermodynamics, the constrained quantity is often mean energy. For microstates with energies , maximizing entropy subject to normalization and yields
The parameter is determined by the energy constraint. In thermal equilibrium it equals , where is absolute temperature and is the Boltzmann constant. The distribution is the canonical ensemble.
The physical entropy is , and the mean energy is
The Helmholtz free energy satisfies . At fixed temperature, minimizing the functional over normalized distributions is equivalent to selecting the canonical distribution. Thus entropy maximization at a specified mean energy and free-energy minimization at a specified temperature are related descriptions, but their externally controlled quantities differ.
If particle number can fluctuate, imposing both mean energy and mean particle number gives the grand canonical form
where is chemical potential and is the grand partition function. If energy is instead fixed exactly and the accessible microstates have equal reference weights, normalization over that energy shell gives the microcanonical ensemble. Degeneracy matters: an energy level containing many microstates generally has greater total probability than a single microstate at that energy.
For quantum mechanics, the state is represented by a density operator . Maximizing von Neumann entropy,
under linear expectation constraints produces an operator exponential. An energy constraint gives . Several constrained observables need not commute; the exponent is then the sum of their weighted operators, not generally a product of separate exponentials.
A related extension, maximum caliber, assigns probabilities to trajectories rather than instantaneous states. It maximizes path entropy, or minimizes path relative entropy, subject to constraints such as average transition counts, integrated currents, or residence times. With a suitable reference path measure and constraints, this produces stochastic dynamics, including Markov descriptions.
An equilibrium entropy maximum is not by itself a derivation of microscopic relaxation. It identifies a distribution compatible with specified information. Explaining how a physical system approaches equilibrium additionally involves dynamics, conservation laws, environmental coupling, and relevant time scales.
Algorithms and practical applications
For a finite state space of manageable size, the dual objective
can be solved using convex optimization. Its gradient is the feature-mean error, and its Hessian is the feature covariance matrix. Newton’s method uses that covariance structure to update parameters, while quasi-Newton and first-order methods avoid some of the cost of forming or solving large Hessian systems. Near-dependent features and constraints close to the attainable boundary can make the computation ill-conditioned.
Numerical evaluation of the partition function requires care. If are log weights, the stable identity
avoids overflow from large exponentials. Stable normalization is especially important when fitted probabilities vary by many orders of magnitude. Optimization stopping rules can examine both objective changes and residual errors in the required moments.
For contingency tables, iterative proportional fitting repeatedly rescales entries to match prescribed margins. Under suitable compatibility and support conditions, it finds the minimum-relative-entropy table relative to an initial table. Beginning with uniform positive entries gives a maximum entropy solution. The method is useful because each adjustment solves a simple subproblem, although structural zeros and incompatible margins require separate treatment.
In conditional classification, maximum entropy models commonly take the form
With appropriate features, this gives logistic regression and multinomial softmax models. Here the constraints involve feature averages over an empirical distribution of inputs. The conditional formulation specifies output uncertainty given each input rather than attempting to model the entire joint distribution of inputs and outputs.
For binary variables, fixing single-variable means and pairwise products yields a distribution of the form
the Ising-model form. Such models have been used to describe collective activity and dependencies. Pairwise constraints do not make all higher-order dependencies vanish; rather, higher-order behavior is whatever the fitted pairwise model entails.
Large systems may have exponentially many configurations, making exact normalization impractical. Monte Carlo methods, including Markov chain Monte Carlo, can estimate feature expectations during fitting, but introduce sampling error and mixing problems. Other applications include positive-image reconstruction and species-distribution modeling. In each case, the reference distribution, observation process, and constraint uncertainty are crucial parts of the model, not details supplied by the entropy objective.
Justifications, scope, and limitations
One justification for maximum entropy comes from counting empirical frequencies. Consider independent draws from a uniform distribution on categories, with counts . The number of sequences having those counts is
Using Stirling’s approximation, its logarithm has leading term . Among frequency patterns satisfying given aggregate restrictions, the entropy maximizer therefore corresponds to the greatest number of underlying sequences. For independent draws from a nonuniform reference , the leading exponential probability instead involves .
This argument makes its assumptions visible. It uses independent sampling, a reference distribution, and an asymptotic approximation. It does not establish that every uncertain physical or social system is generated by such a process. It nevertheless explains why entropy and relative entropy appear so often in concentration phenomena and constrained probability models.
A different approach uses consistency requirements. The Shore–Johnson axioms examine requirements such as uniqueness, coordinate invariance, and consistent treatment of information about separate systems or subsets. Under their formulation and supporting assumptions, these requirements lead to relative-entropy updating. The axioms are not assumption-free consequences of logic; the interpretation of independence and the meaning of admissible information are central to their application.
Representation dependence remains a practical limitation. Dividing one category into several subcategories changes a uniform-reference maximum entropy calculation unless the reference weights are adjusted consistently. Likewise, entropy maximization does not automatically imply statistical independence. If complete marginal distributions are fixed and no dependence constraints are imposed, the joint entropy is maximized by their product. A fixed average interaction, however, generally produces dependence.
Sequential updating requires equal care. If a new constraint supplements an old one, both belong in the final feasible set. An update that retains only the latest condition may erase earlier information. Other situations involve genuinely superseding information rather than accumulating constraints. The optimization procedure cannot decide which interpretation is appropriate.
Finally, maximum entropy is not interchangeable with maximizing thermodynamic entropy production, maximizing disorder informally, or choosing whatever model seems most unpredictable. Nor does an entropy-optimal fit establish causal relationships from observational constraints. The principle solves a sharply specified distribution-selection problem. Questions about experimental validity, omitted variables, sampling bias, and predictive adequacy remain questions about the surrounding scientific model and the evidence used to construct it.