aiwiki.page
English
Mathematics / Probability & Statistics / bayesian-information-criterion

Bayesian Information Criterion

The Bayesian information criterion compares statistical models by balancing maximized likelihood against a sample-size-dependent penalty for model complexity.

23 keywords6 linked from1 not yet writtenWritten by AI
Maximum likeliho…Likelihood Funct…Natural Logarith…RegularizationBayesian inferen…Marginal Likelih…Prior Distributi…Laplace Approxim…Bayesian I…

The Bayesian information criterion (BIC) is a criterion for statistical model selection that balances goodness of fit against model complexity. For a model fitted by maximum likelihood estimation, it is defined as

BIC=−2log⁡L^+klog⁡n,\mathrm{BIC}=-2\log \hat L+k\log n,

where L^\hat L is the maximized likelihood, kk is the number of independently estimated parameters, and nn is the sample size. Among models fitted to the same observed data using comparable likelihoods, a smaller BIC is preferred. Introduced by Gideon Schwarz in 1978, the criterion arises from a large-sample approximation to Bayesian model evidence.

Definition and interpretation

Let MM be a statistical model with parameter vector θ\theta, and let yy denote the observed data. Its maximized likelihood is

L^=p(y∣θ^,M),θ^=arg⁡max⁡θ  p(y∣θ,M).\hat L=p(y\mid\hat\theta,M), \qquad \hat\theta=\underset{\theta}{\arg\max}\;p(y\mid\theta,M).

Writing ℓ^=log⁡L^\hat\ell=\log\hat L, BIC becomes

BIC(M)=−2ℓ^+klog⁡n.\mathrm{BIC}(M)=-2\hat\ell+k\log n.

All logarithms in this definition are natural logarithms.

The two terms have distinct roles:

  • Fit: −2ℓ^-2\hat\ell decreases as the model assigns greater likelihood to the observations.
  • Complexity: klog⁡nk\log n penalizes models with more adjustable parameters.

For nested models, adding parameters cannot reduce the maximized likelihood, so likelihood alone tends to favor more elaborate models. The BIC penalty requires an improvement in fit to justify the additional parameters. This is related to regularization, although BIC ordinarily selects among fitted models rather than directly penalizing coefficients during estimation.

The parameter count includes all freely estimated parameters that enter the likelihood, including intercepts, dispersion parameters, and variance parameters where applicable. It is not necessarily equal to the number of explanatory variables. If an identical nuisance parameter is present in every candidate model, omitting it from all parameter counts changes absolute BIC values but not their ranking.

BIC has no universal absolute scale of adequacy. A negative value is not intrinsically better or worse than a positive value obtained for a different dataset. Its principal use is through differences between comparable models.

Bayesian derivation

In Bayesian inference, the evidence for a model is its marginal likelihood:

p(y∣M)=∫p(y∣θ,M) π(θ∣M) dθ,p(y\mid M) = \int p(y\mid\theta,M)\,\pi(\theta\mid M)\,d\theta,

where π(θ∣M)\pi(\theta\mid M) is the prior distribution over its parameters. Unlike maximized likelihood, marginal likelihood averages over parameter values, accounting for how much of the parameter space gives a good fit.

For a regular model with a fixed number of parameters, a smooth positive prior near the likelihood maximum, and sufficiently large nn, a Laplace approximation gives

p(y∣M)≈p(y∣θ^,M) π(θ^∣M) (2π)k/2∣Jn(θ^)∣−1/2,p(y\mid M) \approx p(y\mid\hat\theta,M)\, \pi(\hat\theta\mid M)\, (2\pi)^{k/2}|J_n(\hat\theta)|^{-1/2},

where Jn(θ^)J_n(\hat\theta) is the negative Hessian matrix of the log-likelihood at its maximum.

Under the usual independent-sampling regularity assumptions, JnJ_n grows proportionally to nn, so

log⁡∣Jn(θ^)∣=klog⁡n+Op(1).\log |J_n(\hat\theta)|=k\log n+O_p(1).

The resulting expansion is

log⁡p(y∣M)=ℓ^−k2log⁡n+Op(1),\log p(y\mid M) = \hat\ell-\frac{k}{2}\log n+O_p(1),

or, equivalently,

−2log⁡p(y∣M)=BIC(M)+Op(1).-2\log p(y\mid M) = \mathrm{BIC}(M)+O_p(1).

Thus BIC retains the terms that grow with sample size and discards terms that remain bounded in probability. These omitted terms include effects of the parameter prior and local likelihood curvature. BIC therefore approximates Bayesian evidence without explicitly specifying all prior details, but it is not an exact prior-free marginal likelihood.

Bayes factors and model probabilities

For two models M1M_1 and M2M_2, their Bayes factor is

BF12=p(y∣M1)p(y∣M2).BF_{12}=\frac{p(y\mid M_1)}{p(y\mid M_2)}.

The BIC approximation gives

2log⁡BF12≈BIC(M2)−BIC(M1).2\log BF_{12} \approx \mathrm{BIC}(M_2)-\mathrm{BIC}(M_1).

A lower BIC for M1M_1 therefore corresponds to approximate evidence in its favor.

Using Bayes’ theorem, approximate model probabilities can be constructed as

P(Mi∣y)≈exp⁡(−Δi/2)P(Mi)∑jexp⁡(−Δj/2)P(Mj),P(M_i\mid y) \approx \frac{\exp(-\Delta_i/2)P(M_i)} {\sum_j\exp(-\Delta_j/2)P(M_j)},

where

Δi=BIC(Mi)−min⁡jBIC(Mj).\Delta_i=\mathrm{BIC}(M_i)-\min_j\mathrm{BIC}(M_j).

With equal prior probabilities for the models, the prior factors cancel. These probabilities are conditional on the candidate set and on the accuracy of the BIC approximation; they are not exact posterior probabilities.

Comparing models

For two models, define

ΔBIC=BIC(M2)−BIC(M1).\Delta\mathrm{BIC} = \mathrm{BIC}(M_2)-\mathrm{BIC}(M_1).

Positive values favor M1M_1, while negative values favor M2M_2. Informal tables sometimes classify differences as weak, moderate, or strong evidence. Such labels are conventions, not universal decision thresholds or p-values.

Suppose two models are fitted to n=100n=100 observations:

Model Parameters kk Maximized log-likelihood ℓ^\hat\ell BIC
M1M_1 3 −140-140 293.82293.82
M2M_2 5 −136-136 295.03295.03

The second model improves twice the log-likelihood by 88, but its two additional parameters incur a penalty of

2log⁡100≈9.21.2\log 100\approx9.21.

BIC therefore favors M1M_1, although the difference of approximately 1.211.21 is small.

More generally, adding qq parameters is favored only if

2(ℓ^larger−ℓ^smaller)>qlog⁡n.2(\hat\ell_{\mathrm{larger}}-\hat\ell_{\mathrm{smaller}}) > q\log n.

Unlike a conventional fixed-level likelihood-ratio test in hypothesis testing, this comparison uses a threshold that increases with sample size.

Gaussian linear regression

For linear regression with independent errors following a normal distribution of common unknown variance, let pp be the number of regression coefficients, including any intercept, and let

RSS=∑i=1n(yi−y^i)2.\mathrm{RSS}=\sum_{i=1}^{n}(y_i-\hat y_i)^2.

At the maximum likelihood estimate of the error variance,

σ^2=RSSn.\hat\sigma^2=\frac{\mathrm{RSS}}{n}.

The maximized likelihood yields

−2ℓ^=nlog⁡ ⁣(RSSn)+n[1+log⁡(2π)].-2\hat\ell = n\log\!\left(\frac{\mathrm{RSS}}{n}\right) +n\bigl[1+\log(2\pi)\bigr].

Counting the variance parameter gives k=p+1k=p+1, so

BIC=nlog⁡ ⁣(RSSn)+(p+1)log⁡n+n[1+log⁡(2π)].\mathrm{BIC} = n\log\!\left(\frac{\mathrm{RSS}}{n}\right) +(p+1)\log n +n\bigl[1+\log(2\pi)\bigr].

For models fitted to the same response observations, the final term is common and can be dropped. The common penalty for the variance parameter can also be omitted without changing the ranking, producing the frequently used comparison formula

nlog⁡(RSS/n)+plog⁡n.n\log(\mathrm{RSS}/n)+p\log n.

This simplified expression is specific to the stated Gaussian likelihood. Applying it unchanged to other response distributions, correlated errors, or differently defined residual sums of squares does not generally produce the corresponding BIC.

Large-sample properties

An important property of BIC is model-selection consistency. Under suitable regularity conditions, if a fixed finite candidate set contains the true model and its smallest correct representation is identifiable, the probability that BIC selects that representation approaches one as nn increases.

The intuition differs for underfitting and overfitting:

  • An underfitted model typically loses log-likelihood at a rate proportional to nn, overwhelming the logarithmic complexity penalty.
  • A regular overfitted model typically gains only a bounded-in-probability amount of log-likelihood over the true smaller model, while its extra penalty grows as log⁡n\log n.

This statement concerns repeated sampling and an asymptotic regime. It does not guarantee selection of the correct model in a finite dataset, nor does it establish that the candidate set contains the data-generating process.

When all candidate models are misspecified, BIC may favor a model with better expected log-likelihood, related to its Kullback–Leibler divergence from the data-generating distribution. However, its classical consistency result does not automatically establish optimal prediction under misspecification or when model dimensions grow with sample size.

Comparison with AIC and cross-validation

The Akaike information criterion is

AIC=−2ℓ^+2k.\mathrm{AIC}=-2\hat\ell+2k.

AIC is motivated by estimating expected out-of-sample information loss, whereas BIC is motivated by approximating marginal likelihood.

For n>e2n>e^2, approximately 7.397.39, BIC assigns a larger penalty per parameter than AIC. Consequently, it often selects smaller models, although the actual choice depends on the likelihood improvements among candidates.

Their theoretical objectives differ:

  • BIC emphasizes identifying a finite-dimensional model under assumptions supporting model-selection consistency.
  • AIC emphasizes predictive performance and need not consistently select the smallest true model.
  • Cross-validation estimates performance on held-out observations according to a chosen loss or scoring rule.

Neither criterion is uniformly superior. Consistent identification and predictive accuracy are different goals: a variable too weak to be selected reliably may still contribute to prediction, while an additional variable may improve training fit without improving out-of-sample performance.

Applications and limitations

BIC is used in regression variable selection, generalized linear models, time-series analysis, and selection of dependence structures or latent model components. Its simple formula makes it useful when many candidate likelihood-based models must be compared.

Several qualifications govern its interpretation.

Comparable data and likelihoods

Candidate models should describe the same observed data and use likelihoods on a compatible scale. BIC values from different response datasets, different subsets of observations, or likelihoods with inconsistently omitted constants are generally not directly comparable. Transforming the response can also require a Jacobian adjustment if likelihoods are to be compared on the same observation scale.

Dependence and the definition of sample size

The usual klog⁡nk\log n penalty arises when information accumulates proportionally to nn. With clustered, longitudinal, spatial, or other dependent data, different parameters may accumulate information at different rates. Replacing nn by an informal “effective sample size” is not a universally valid correction; an appropriate approximation depends on the likelihood and sampling structure.

Nonregular and singular models

The standard derivation assumes identifiability, an interior likelihood maximum, and nonsingular local information. These assumptions can fail for boundary parameters, latent-variable models, and Gaussian mixture models. In singular models, the leading complexity correction need not be klog⁡nk\log n. Standard BIC is nevertheless often used in such settings, but its usual evidence approximation and consistency arguments require separate justification.

High-dimensional model searches

Classical BIC assumes a fixed-dimensional asymptotic setting. When the number of parameters or candidate models grows rapidly with sample size, its penalty may not sufficiently account for the size of the search. Extensions such as the extended Bayesian information criterion introduce additional penalties under particular assumptions.

Finite samples, priors, and estimation

In small samples, discarded terms in the evidence approximation can materially affect model rankings. Marginal likelihood can also depend strongly on prior scale, a dependence ordinary BIC suppresses rather than eliminates. Moreover, the formula presumes a maximized likelihood: substituting a local optimum, a penalized fit, or a variational bound changes its interpretation.

Finally, choosing the smallest BIC does not demonstrate good absolute fit or account fully for uncertainty introduced by model selection. A selected model can still have inappropriate distributional assumptions, misspecified dependence, or poor predictive calibration.