The Bayesian information criterion (BIC) is a criterion for statistical model selection that balances goodness of fit against model complexity. For a model fitted by maximum likelihood estimation, it is defined as
where is the maximized likelihood, is the number of independently estimated parameters, and is the sample size. Among models fitted to the same observed data using comparable likelihoods, a smaller BIC is preferred. Introduced by Gideon Schwarz in 1978, the criterion arises from a large-sample approximation to Bayesian model evidence.
Definition and interpretation
Let be a statistical model with parameter vector , and let denote the observed data. Its maximized likelihood is
Writing , BIC becomes
All logarithms in this definition are natural logarithms.
The two terms have distinct roles:
- Fit: decreases as the model assigns greater likelihood to the observations.
- Complexity: penalizes models with more adjustable parameters.
For nested models, adding parameters cannot reduce the maximized likelihood, so likelihood alone tends to favor more elaborate models. The BIC penalty requires an improvement in fit to justify the additional parameters. This is related to regularization, although BIC ordinarily selects among fitted models rather than directly penalizing coefficients during estimation.
The parameter count includes all freely estimated parameters that enter the likelihood, including intercepts, dispersion parameters, and variance parameters where applicable. It is not necessarily equal to the number of explanatory variables. If an identical nuisance parameter is present in every candidate model, omitting it from all parameter counts changes absolute BIC values but not their ranking.
BIC has no universal absolute scale of adequacy. A negative value is not intrinsically better or worse than a positive value obtained for a different dataset. Its principal use is through differences between comparable models.
Bayesian derivation
In Bayesian inference, the evidence for a model is its marginal likelihood:
where is the prior distribution over its parameters. Unlike maximized likelihood, marginal likelihood averages over parameter values, accounting for how much of the parameter space gives a good fit.
For a regular model with a fixed number of parameters, a smooth positive prior near the likelihood maximum, and sufficiently large , a Laplace approximation gives
where is the negative Hessian matrix of the log-likelihood at its maximum.
Under the usual independent-sampling regularity assumptions, grows proportionally to , so
The resulting expansion is
or, equivalently,
Thus BIC retains the terms that grow with sample size and discards terms that remain bounded in probability. These omitted terms include effects of the parameter prior and local likelihood curvature. BIC therefore approximates Bayesian evidence without explicitly specifying all prior details, but it is not an exact prior-free marginal likelihood.
Bayes factors and model probabilities
For two models and , their Bayes factor is
The BIC approximation gives
A lower BIC for therefore corresponds to approximate evidence in its favor.
Using Bayes’ theorem, approximate model probabilities can be constructed as
where
With equal prior probabilities for the models, the prior factors cancel. These probabilities are conditional on the candidate set and on the accuracy of the BIC approximation; they are not exact posterior probabilities.
Comparing models
For two models, define
Positive values favor , while negative values favor . Informal tables sometimes classify differences as weak, moderate, or strong evidence. Such labels are conventions, not universal decision thresholds or p-values.
Suppose two models are fitted to observations:
| Model | Parameters | Maximized log-likelihood | BIC |
|---|---|---|---|
| 3 | |||
| 5 |
The second model improves twice the log-likelihood by , but its two additional parameters incur a penalty of
BIC therefore favors , although the difference of approximately is small.
More generally, adding parameters is favored only if
Unlike a conventional fixed-level likelihood-ratio test in hypothesis testing, this comparison uses a threshold that increases with sample size.
Gaussian linear regression
For linear regression with independent errors following a normal distribution of common unknown variance, let be the number of regression coefficients, including any intercept, and let
At the maximum likelihood estimate of the error variance,
The maximized likelihood yields
Counting the variance parameter gives , so
For models fitted to the same response observations, the final term is common and can be dropped. The common penalty for the variance parameter can also be omitted without changing the ranking, producing the frequently used comparison formula
This simplified expression is specific to the stated Gaussian likelihood. Applying it unchanged to other response distributions, correlated errors, or differently defined residual sums of squares does not generally produce the corresponding BIC.
Large-sample properties
An important property of BIC is model-selection consistency. Under suitable regularity conditions, if a fixed finite candidate set contains the true model and its smallest correct representation is identifiable, the probability that BIC selects that representation approaches one as increases.
The intuition differs for underfitting and overfitting:
- An underfitted model typically loses log-likelihood at a rate proportional to , overwhelming the logarithmic complexity penalty.
- A regular overfitted model typically gains only a bounded-in-probability amount of log-likelihood over the true smaller model, while its extra penalty grows as .
This statement concerns repeated sampling and an asymptotic regime. It does not guarantee selection of the correct model in a finite dataset, nor does it establish that the candidate set contains the data-generating process.
When all candidate models are misspecified, BIC may favor a model with better expected log-likelihood, related to its Kullback–Leibler divergence from the data-generating distribution. However, its classical consistency result does not automatically establish optimal prediction under misspecification or when model dimensions grow with sample size.
Comparison with AIC and cross-validation
The Akaike information criterion is
AIC is motivated by estimating expected out-of-sample information loss, whereas BIC is motivated by approximating marginal likelihood.
For , approximately , BIC assigns a larger penalty per parameter than AIC. Consequently, it often selects smaller models, although the actual choice depends on the likelihood improvements among candidates.
Their theoretical objectives differ:
- BIC emphasizes identifying a finite-dimensional model under assumptions supporting model-selection consistency.
- AIC emphasizes predictive performance and need not consistently select the smallest true model.
- Cross-validation estimates performance on held-out observations according to a chosen loss or scoring rule.
Neither criterion is uniformly superior. Consistent identification and predictive accuracy are different goals: a variable too weak to be selected reliably may still contribute to prediction, while an additional variable may improve training fit without improving out-of-sample performance.
Applications and limitations
BIC is used in regression variable selection, generalized linear models, time-series analysis, and selection of dependence structures or latent model components. Its simple formula makes it useful when many candidate likelihood-based models must be compared.
Several qualifications govern its interpretation.
Comparable data and likelihoods
Candidate models should describe the same observed data and use likelihoods on a compatible scale. BIC values from different response datasets, different subsets of observations, or likelihoods with inconsistently omitted constants are generally not directly comparable. Transforming the response can also require a Jacobian adjustment if likelihoods are to be compared on the same observation scale.
Dependence and the definition of sample size
The usual penalty arises when information accumulates proportionally to . With clustered, longitudinal, spatial, or other dependent data, different parameters may accumulate information at different rates. Replacing by an informal “effective sample size” is not a universally valid correction; an appropriate approximation depends on the likelihood and sampling structure.
Nonregular and singular models
The standard derivation assumes identifiability, an interior likelihood maximum, and nonsingular local information. These assumptions can fail for boundary parameters, latent-variable models, and Gaussian mixture models. In singular models, the leading complexity correction need not be . Standard BIC is nevertheless often used in such settings, but its usual evidence approximation and consistency arguments require separate justification.
High-dimensional model searches
Classical BIC assumes a fixed-dimensional asymptotic setting. When the number of parameters or candidate models grows rapidly with sample size, its penalty may not sufficiently account for the size of the search. Extensions such as the extended Bayesian information criterion introduce additional penalties under particular assumptions.
Finite samples, priors, and estimation
In small samples, discarded terms in the evidence approximation can materially affect model rankings. Marginal likelihood can also depend strongly on prior scale, a dependence ordinary BIC suppresses rather than eliminates. Moreover, the formula presumes a maximized likelihood: substituting a local optimum, a penalized fit, or a variational bound changes its interpretation.
Finally, choosing the smallest BIC does not demonstrate good absolute fit or account fully for uncertainty introduced by model selection. A selected model can still have inappropriate distributional assumptions, misspecified dependence, or poor predictive calibration.