The Dirichlet distribution is a family of continuous probability distributions for vectors of proportions whose components are nonnegative and sum to one. It generalizes the beta distribution from two components to any finite number of components. Its parameters control both the expected proportions and the concentration of probability around them. In Bayesian inference, it is especially important as a conjugate prior for categorical and multinomial probabilities.
Definition and density
Let be an integer, and let
A random vector has a Dirichlet distribution, written
if its probability density is
on the interior of the probability simplex
The normalizing constant is the multivariate beta function,
where is the gamma function.
The density is conventionally taken with respect to the coordinates , with
It is not a density with respect to -dimensional volume: the simplex lies in a -dimensional hyperplane. A density defined instead relative to Euclidean surface measure requires a constant adjustment.
For strictly positive parameters, all components are strictly positive almost surely. Boundary points have probability zero, even when the density becomes unbounded as a component approaches zero.
When , the first component satisfies
Parameter interpretation and shape
A useful parameterization separates the mean vector from the total concentration:
The vector specifies the expected proportions, while controls their dispersion. Holding fixed, increasing concentrates the distribution around ; decreasing spreads probability toward the boundary.
The individual parameters determine boundary behavior:
- If , the density tends to zero as approaches zero while the other coordinates remain away from their boundaries.
- If , the corresponding power of is zero.
- If , the density diverges as approaches zero.
A symmetric Dirichlet distribution has for every component. Its mean is the uniform vector .
- For , the distribution is uniform over the simplex.
- For , it favors relatively balanced vectors.
- For , it favors vectors near the boundary, often with a few large components and many small ones.
“Uniform over the simplex” does not mean that each coordinate is uniformly distributed. For , a coordinate under has distribution .
The distinction between and the total concentration matters when comparing models with different numbers of categories.
Moments and dependence
Writing , the mean and variance are
For distinct components,
Thus every pair of distinct components has negative covariance. The covariance matrix can be written compactly as
It is singular because the coordinates sum exactly to one.
More generally, for real exponents satisfying ,
The expected log proportions are
where is the digamma function. These expectations appear in parameter estimation and variational inference.
If every , the density has a unique interior mode:
This formula is not a general mode formula when some parameters are at most one. In those cases, density maxima or divergent limiting values can occur at the boundary, and modes may be nonunique.
Marginals, aggregation, and conditioning
The Dirichlet family has several closure properties that make it convenient for modeling proportions.
Marginal distributions
Each coordinate has a beta distribution:
More generally, for a nonempty proper subset of the component indices,
Aggregation
If partition the component indices and
then
Merging categories therefore corresponds to adding their parameters.
Normalized subvectors
For a subset , let . The normalized subvector satisfies
It is independent of and of the components outside . This strong independence structure is often described through the Dirichlet distribution’s neutrality properties.
Gamma construction and simulation
A fundamental construction uses independent gamma random variables. Let
independently, with a common rate . Then
has distribution .
Moreover,
and is independent of . The common rate disappears on normalization.
This gives a standard simulation method: generate independent gamma variates and normalize their sum. For very small shape parameters, implementations require care because tiny gamma variates can underflow in floating-point arithmetic.
An alternative is a finite stick-breaking construction. Independently draw
and set
Each determines the fraction of the remaining mass allocated to component .
Bayesian inference for categorical data
Suppose is an unknown category-probability vector. Given counts , with , the multinomial distribution gives a likelihood proportional to
If the prior distribution is
multiplication by this likelihood yields the posterior distribution
This is the conjugacy relationship.
The posterior mean is
For , it can also be written as
a weighted average of the prior mean and the observed frequency. This motivates interpreting the as prior pseudocounts, although that interpretation depends on which posterior summary is being considered.
For a new observation from a categorical distribution, the posterior predictive probability is
For example, a prior and observed counts give a posterior and predictive probabilities .
Integrating out gives the Dirichlet–multinomial distribution:
It allows greater count variability than a multinomial model with fixed probabilities.
The all-ones prior is uniform over the simplex, but it is not uniquely noninformative. For the multinomial model, the Jeffreys prior is . Taking every parameter to zero does not produce a proper Dirichlet distribution.
Estimating the parameters
Given independent observed proportion vectors with strictly positive components, the log-likelihood is
The maximum-likelihood equations are
They generally require numerical solution, using methods such as Newton’s method or fixed-point iteration. The constraints must be preserved.
Observed exact zeros require separate treatment: the ordinary Dirichlet model assigns them probability zero, and their logarithms are undefined. Zeros caused by rounding or finite sampling differ from structural zeros that represent genuinely impossible components.
Applications and limitations
Dirichlet distributions are used for uncertain class probabilities, mixture weights, transition probabilities, and compositional proportions. In latent Dirichlet allocation, they commonly provide priors for document–topic and topic–word probability vectors. The Dirichlet process extends the finite-partition structure: probabilities assigned to the sets in any finite measurable partition have a Dirichlet distribution.
Despite its tractability, the family imposes substantial restrictions:
- Negative pairwise covariance. Every pair of distinct components is negatively correlated. The sum-to-one constraint alone does not require this; other distributions on a simplex can permit positive association between selected components.
- Restricted dispersion structure. Once the mean is fixed, a single total concentration controls all component variances.
- No exact zeros. Small parameters can generate very small proportions, but not positive probability at zero.
- Limited multimodal flexibility. A single Dirichlet distribution cannot describe arbitrary clusters of compositions centered at several unrelated interior locations.
Alternatives include mixtures of Dirichlet distributions, explicit models for structural zeros, and the logistic-normal distribution, which transforms a multivariate normal vector into proportions and supports richer dependence structures.