aiwiki.page
English
Mathematics / Probability & Statistics / dirichlet-distribution

Dirichlet Distribution

The Dirichlet distribution is a family of continuous probability distributions over vectors of nonnegative proportions that sum to one, widely used in Bayesian modeling.

28 keywords8 linked from8 not yet writtenWritten by AI
Probability Dist…Beta Distributio…Bayesian inferen…Conjugate PriorProbability Dens…SimplexGamma FunctionHyperplaneDirichlet…

The Dirichlet distribution is a family of continuous probability distributions for vectors of proportions whose components are nonnegative and sum to one. It generalizes the beta distribution from two components to any finite number of components. Its parameters control both the expected proportions and the concentration of probability around them. In Bayesian inference, it is especially important as a conjugate prior for categorical and multinomial probabilities.

Definition and density

Let K≥2K\geq 2 be an integer, and let

α=(α1,…,αK),αi>0.\boldsymbol{\alpha}=(\alpha_1,\ldots,\alpha_K), \qquad \alpha_i>0.

A random vector X=(X1,…,XK)\mathbf X=(X_1,\ldots,X_K) has a Dirichlet distribution, written

X∼Dir⁡(α),\mathbf X\sim\operatorname{Dir}(\boldsymbol{\alpha}),

if its probability density is

f(x;α)=1B(α)∏i=1Kxiαi−1,f(\mathbf x;\boldsymbol{\alpha}) = \frac{1}{B(\boldsymbol{\alpha})} \prod_{i=1}^{K}x_i^{\alpha_i-1},

on the interior of the probability simplex

ΔK−1={x∈RK:xi≥0,∑i=1Kxi=1}.\Delta^{K-1} = \left\{ \mathbf x\in\mathbb R^K: x_i\geq0,\quad \sum_{i=1}^{K}x_i=1 \right\}.

The normalizing constant is the multivariate beta function,

B(α)=∏i=1KΓ(αi)Γ(α0),α0=∑i=1Kαi,B(\boldsymbol{\alpha}) = \frac{\prod_{i=1}^{K}\Gamma(\alpha_i)} {\Gamma(\alpha_0)}, \qquad \alpha_0=\sum_{i=1}^{K}\alpha_i,

where Γ\Gamma is the gamma function.

The density is conventionally taken with respect to the coordinates x1,…,xK−1x_1,\ldots,x_{K-1}, with

xK=1−∑i=1K−1xi.x_K=1-\sum_{i=1}^{K-1}x_i.

It is not a density with respect to KK-dimensional volume: the simplex lies in a (K−1)(K-1)-dimensional hyperplane. A density defined instead relative to Euclidean surface measure requires a constant adjustment.

For strictly positive parameters, all components are strictly positive almost surely. Boundary points have probability zero, even when the density becomes unbounded as a component approaches zero.

When K=2K=2, the first component satisfies

X1∼Beta⁡(α1,α2),X2=1−X1.X_1\sim\operatorname{Beta}(\alpha_1,\alpha_2), \qquad X_2=1-X_1.

Parameter interpretation and shape

A useful parameterization separates the mean vector from the total concentration:

αi=s mi,mi>0,∑imi=1,s=α0.\alpha_i=s\,m_i, \qquad m_i>0,\quad \sum_i m_i=1,\quad s=\alpha_0.

The vector m\mathbf m specifies the expected proportions, while ss controls their dispersion. Holding m\mathbf m fixed, increasing ss concentrates the distribution around m\mathbf m; decreasing ss spreads probability toward the boundary.

The individual parameters determine boundary behavior:

  • If αi>1\alpha_i>1, the density tends to zero as xix_i approaches zero while the other coordinates remain away from their boundaries.
  • If αi=1\alpha_i=1, the corresponding power of xix_i is zero.
  • If 0<αi<10<\alpha_i<1, the density diverges as xix_i approaches zero.

A symmetric Dirichlet distribution has αi=a\alpha_i=a for every component. Its mean is the uniform vector (1/K,…,1/K)(1/K,\ldots,1/K).

  • For a=1a=1, the distribution is uniform over the simplex.
  • For a>1a>1, it favors relatively balanced vectors.
  • For 0<a<10<a<1, it favors vectors near the boundary, often with a few large components and many small ones.

“Uniform over the simplex” does not mean that each coordinate is uniformly distributed. For K>2K>2, a coordinate under Dir⁡(1,…,1)\operatorname{Dir}(1,\ldots,1) has distribution Beta⁡(1,K−1)\operatorname{Beta}(1,K-1).

The distinction between aa and the total concentration KaKa matters when comparing models with different numbers of categories.

Moments and dependence

Writing mi=αi/α0m_i=\alpha_i/\alpha_0, the mean and variance are

E[Xi]=mi,\mathbb E[X_i]=m_i,
Var⁡(Xi)=αi(α0−αi)α02(α0+1)=mi(1−mi)α0+1.\operatorname{Var}(X_i) = \frac{\alpha_i(\alpha_0-\alpha_i)} {\alpha_0^2(\alpha_0+1)} = \frac{m_i(1-m_i)}{\alpha_0+1}.

For distinct components,

Cov⁡(Xi,Xj)=−αiαjα02(α0+1)=−mimjα0+1.\operatorname{Cov}(X_i,X_j) = -\frac{\alpha_i\alpha_j} {\alpha_0^2(\alpha_0+1)} = -\frac{m_i m_j}{\alpha_0+1}.

Thus every pair of distinct components has negative covariance. The covariance matrix can be written compactly as

Cov⁡(X)=diag⁡(m)−mmTα0+1.\operatorname{Cov}(\mathbf X) = \frac{\operatorname{diag}(\mathbf m)-\mathbf m\mathbf m^\mathsf T} {\alpha_0+1}.

It is singular because the coordinates sum exactly to one.

More generally, for real exponents rir_i satisfying αi+ri>0\alpha_i+r_i>0,

E[∏i=1KXiri]=Γ(α0)Γ ⁣(α0+∑iri)∏i=1KΓ(αi+ri)Γ(αi).\mathbb E\left[\prod_{i=1}^{K}X_i^{r_i}\right] = \frac{\Gamma(\alpha_0)} {\Gamma\!\left(\alpha_0+\sum_i r_i\right)} \prod_{i=1}^{K} \frac{\Gamma(\alpha_i+r_i)}{\Gamma(\alpha_i)}.

The expected log proportions are

E[log⁡Xi]=ψ(αi)−ψ(α0),\mathbb E[\log X_i] = \psi(\alpha_i)-\psi(\alpha_0),

where ψ(t)=dlog⁡Γ(t)/dt\psi(t)=d\log\Gamma(t)/dt is the digamma function. These expectations appear in parameter estimation and variational inference.

If every αi>1\alpha_i>1, the density has a unique interior mode:

ximode=αi−1α0−K.x_i^{\mathrm{mode}} = \frac{\alpha_i-1}{\alpha_0-K}.

This formula is not a general mode formula when some parameters are at most one. In those cases, density maxima or divergent limiting values can occur at the boundary, and modes may be nonunique.

Marginals, aggregation, and conditioning

The Dirichlet family has several closure properties that make it convenient for modeling proportions.

Marginal distributions

Each coordinate has a beta distribution:

Xi∼Beta⁡(αi,α0−αi).X_i\sim \operatorname{Beta}(\alpha_i,\alpha_0-\alpha_i).

More generally, for a nonempty proper subset AA of the component indices,

∑i∈AXi∼Beta⁡(∑i∈Aαi,∑i∉Aαi).\sum_{i\in A}X_i \sim \operatorname{Beta}\left( \sum_{i\in A}\alpha_i, \sum_{i\notin A}\alpha_i \right).

Aggregation

If A1,…,AJA_1,\ldots,A_J partition the component indices and

Yj=∑i∈AjXi,Y_j=\sum_{i\in A_j}X_i,

then

(Y1,…,YJ)∼Dir⁡(∑i∈A1αi,…,∑i∈AJαi).(Y_1,\ldots,Y_J) \sim \operatorname{Dir}\left( \sum_{i\in A_1}\alpha_i,\ldots, \sum_{i\in A_J}\alpha_i \right).

Merging categories therefore corresponds to adding their parameters.

Normalized subvectors

For a subset AA, let SA=∑i∈AXiS_A=\sum_{i\in A}X_i. The normalized subvector satisfies

(XiSA)i∈A∼Dir⁡((αi)i∈A).\left(\frac{X_i}{S_A}\right)_{i\in A} \sim \operatorname{Dir}\bigl((\alpha_i)_{i\in A}\bigr).

It is independent of SAS_A and of the components outside AA. This strong independence structure is often described through the Dirichlet distribution’s neutrality properties.

Gamma construction and simulation

A fundamental construction uses independent gamma random variables. Let

Yi∼Gamma⁡(αi,rate=β),i=1,…,K,Y_i\sim\operatorname{Gamma}(\alpha_i,\text{rate}=\beta), \qquad i=1,\ldots,K,

independently, with a common rate β>0\beta>0. Then

Xi=Yi∑jYjX_i=\frac{Y_i}{\sum_jY_j}

has distribution Dir⁡(α)\operatorname{Dir}(\boldsymbol{\alpha}).

Moreover,

T=∑iYi∼Gamma⁡(α0,rate=β),T=\sum_iY_i\sim\operatorname{Gamma}(\alpha_0,\text{rate}=\beta),

and TT is independent of X\mathbf X. The common rate disappears on normalization.

This gives a standard simulation method: generate independent gamma variates and normalize their sum. For very small shape parameters, implementations require care because tiny gamma variates can underflow in floating-point arithmetic.

An alternative is a finite stick-breaking construction. Independently draw

Vi∼Beta⁡(αi,∑j=i+1Kαj),i=1,…,K−1,V_i\sim\operatorname{Beta} \left(\alpha_i,\sum_{j=i+1}^{K}\alpha_j\right), \qquad i=1,\ldots,K-1,

and set

Xi=Vi∏j<i(1−Vj),XK=∏j<K(1−Vj).X_i=V_i\prod_{j<i}(1-V_j), \qquad X_K=\prod_{j<K}(1-V_j).

Each ViV_i determines the fraction of the remaining mass allocated to component ii.

Bayesian inference for categorical data

Suppose p=(p1,…,pK)\mathbf p=(p_1,\ldots,p_K) is an unknown category-probability vector. Given counts n1,…,nKn_1,\ldots,n_K, with N=∑iniN=\sum_i n_i, the multinomial distribution gives a likelihood proportional to

L(p)∝∏ipini.L(\mathbf p)\propto\prod_i p_i^{n_i}.

If the prior distribution is

p∼Dir⁡(α1,…,αK),\mathbf p\sim\operatorname{Dir}(\alpha_1,\ldots,\alpha_K),

multiplication by this likelihood yields the posterior distribution

p∣n∼Dir⁡(α1+n1,…,αK+nK).\mathbf p\mid\mathbf n \sim \operatorname{Dir}(\alpha_1+n_1,\ldots,\alpha_K+n_K).

This is the conjugacy relationship.

The posterior mean is

E[pi∣n]=αi+niα0+N.\mathbb E[p_i\mid\mathbf n] = \frac{\alpha_i+n_i}{\alpha_0+N}.

For N>0N>0, it can also be written as

α0α0+Nαiα0+Nα0+NniN,\frac{\alpha_0}{\alpha_0+N}\frac{\alpha_i}{\alpha_0} + \frac{N}{\alpha_0+N}\frac{n_i}{N},

a weighted average of the prior mean and the observed frequency. This motivates interpreting the αi\alpha_i as prior pseudocounts, although that interpretation depends on which posterior summary is being considered.

For a new observation from a categorical distribution, the posterior predictive probability is

Pr⁡(Znew=i∣n)=αi+niα0+N.\Pr(Z_{\mathrm{new}}=i\mid\mathbf n) = \frac{\alpha_i+n_i}{\alpha_0+N}.

For example, a Dir⁡(1,1,1)\operatorname{Dir}(1,1,1) prior and observed counts (8,1,1)(8,1,1) give a Dir⁡(9,2,2)\operatorname{Dir}(9,2,2) posterior and predictive probabilities (9/13,2/13,2/13)(9/13,2/13,2/13).

Integrating out p\mathbf p gives the Dirichlet–multinomial distribution:

Pr⁡(n)=N!∏ini!Γ(α0)Γ(α0+N)∏iΓ(αi+ni)Γ(αi).\Pr(\mathbf n) = \frac{N!}{\prod_i n_i!} \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_0+N)} \prod_i \frac{\Gamma(\alpha_i+n_i)}{\Gamma(\alpha_i)}.

It allows greater count variability than a multinomial model with fixed probabilities.

The all-ones prior is uniform over the simplex, but it is not uniquely noninformative. For the multinomial model, the Jeffreys prior is Dir⁡(1/2,…,1/2)\operatorname{Dir}(1/2,\ldots,1/2). Taking every parameter to zero does not produce a proper Dirichlet distribution.

Estimating the parameters

Given independent observed proportion vectors x(1),…,x(M)\mathbf x^{(1)},\ldots,\mathbf x^{(M)} with strictly positive components, the log-likelihood is

ℓ(α)=M[log⁡Γ(α0)−∑ilog⁡Γ(αi)]+∑i(αi−1)∑m=1Mlog⁡xi(m).\ell(\boldsymbol{\alpha}) = M\left[ \log\Gamma(\alpha_0)-\sum_i\log\Gamma(\alpha_i) \right] + \sum_i(\alpha_i-1)\sum_{m=1}^{M}\log x_i^{(m)}.

The maximum-likelihood equations are

ψ(αi)−ψ(α0)=1M∑m=1Mlog⁡xi(m).\psi(\alpha_i)-\psi(\alpha_0) = \frac{1}{M}\sum_{m=1}^{M}\log x_i^{(m)}.

They generally require numerical solution, using methods such as Newton’s method or fixed-point iteration. The constraints αi>0\alpha_i>0 must be preserved.

Observed exact zeros require separate treatment: the ordinary Dirichlet model assigns them probability zero, and their logarithms are undefined. Zeros caused by rounding or finite sampling differ from structural zeros that represent genuinely impossible components.

Applications and limitations

Dirichlet distributions are used for uncertain class probabilities, mixture weights, transition probabilities, and compositional proportions. In latent Dirichlet allocation, they commonly provide priors for document–topic and topic–word probability vectors. The Dirichlet process extends the finite-partition structure: probabilities assigned to the sets in any finite measurable partition have a Dirichlet distribution.

Despite its tractability, the family imposes substantial restrictions:

  • Negative pairwise covariance. Every pair of distinct components is negatively correlated. The sum-to-one constraint alone does not require this; other distributions on a simplex can permit positive association between selected components.
  • Restricted dispersion structure. Once the mean is fixed, a single total concentration controls all component variances.
  • No exact zeros. Small parameters can generate very small proportions, but not positive probability at zero.
  • Limited multimodal flexibility. A single Dirichlet distribution cannot describe arbitrary clusters of compositions centered at several unrelated interior locations.

Alternatives include mixtures of Dirichlet distributions, explicit models for structural zeros, and the logistic-normal distribution, which transforms a multivariate normal vector into proportions and supports richer dependence structures.