{
  "id": "bayesian-information-criterion",
  "title": "Bayesian Information Criterion",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Bayesian information criterion compares statistical models by balancing maximized likelihood against a sample-size-dependent penalty for model complexity.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "BIC",
    "Schwarz criterion",
    "Schwarz information criterion",
    "SIC",
    "SBC"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/bayesian-information-criterion/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/bayesian-information-criterion/",
      "json": "https://aiwiki.page/zh/bayesian-information-criterion.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Bayesian information criterion (BIC) is a criterion for statistical model selection that balances goodness of fit against model complexity. For a model fitted by maximum likelihood estimation, it is defined as\n\n\\[\\mathrm{BIC}=-2\\log \\hat L+k\\log n,\\]\n\nwhere \\(\\hat L\\) is the maximized likelihood, \\(k\\) is the number of independently estimated parameters, and \\(n\\) is the sample size. Among models fitted to the same observed data using comparable likelihoods, a smaller BIC is preferred. Introduced by Gideon Schwarz in 1978, the criterion arises from a large-sample approximation to Bayesian model evidence."
    },
    {
      "heading": "Definition and interpretation",
      "text": "Let \\(M\\) be a statistical model with parameter vector \\(\\theta\\), and let \\(y\\) denote the observed data. Its maximized likelihood is\n\n\\[\\hat L=p(y\\mid\\hat\\theta,M), \\qquad \\hat\\theta=\\underset{\\theta}{\\arg\\max}\\;p(y\\mid\\theta,M).\\]\n\nWriting \\(\\hat\\ell=\\log\\hat L\\), BIC becomes\n\n\\[\\mathrm{BIC}(M)=-2\\hat\\ell+k\\log n.\\]\n\nAll logarithms in this definition are natural logarithms.\n\nThe two terms have distinct roles:\n\n- Fit: \\(-2\\hat\\ell\\) decreases as the model assigns greater likelihood to the observations.\n- Complexity: \\(k\\log n\\) penalizes models with more adjustable parameters.\n\nFor nested models, adding parameters cannot reduce the maximized likelihood, so likelihood alone tends to favor more elaborate models. The BIC penalty requires an improvement in fit to justify the additional parameters. This is related to regularization, although BIC ordinarily selects among fitted models rather than directly penalizing coefficients during estimation.\n\nThe parameter count includes all freely estimated parameters that enter the likelihood, including intercepts, dispersion parameters, and variance parameters where applicable. It is not necessarily equal to the number of explanatory variables. If an identical nuisance parameter is present in every candidate model, omitting it from all parameter counts changes absolute BIC values but not their ranking.\n\nBIC has no universal absolute scale of adequacy. A negative value is not intrinsically better or worse than a positive value obtained for a different dataset. Its principal use is through differences between comparable models."
    },
    {
      "heading": "Bayesian derivation",
      "text": "In Bayesian inference, the evidence for a model is its marginal likelihood:\n\n\\[p(y\\mid M) = \\int p(y\\mid\\theta,M)\\,\\pi(\\theta\\mid M)\\,d\\theta,\\]\n\nwhere \\(\\pi(\\theta\\mid M)\\) is the prior distribution over its parameters. Unlike maximized likelihood, marginal likelihood averages over parameter values, accounting for how much of the parameter space gives a good fit.\n\nFor a regular model with a fixed number of parameters, a smooth positive prior near the likelihood maximum, and sufficiently large \\(n\\), a Laplace approximation gives\n\n\\[p(y\\mid M) \\approx p(y\\mid\\hat\\theta,M)\\, \\pi(\\hat\\theta\\mid M)\\, (2\\pi)^{k/2}|J_n(\\hat\\theta)|^{-1/2},\\]\n\nwhere \\(J_n(\\hat\\theta)\\) is the negative Hessian matrix of the log-likelihood at its maximum.\n\nUnder the usual independent-sampling regularity assumptions, \\(J_n\\) grows proportionally to \\(n\\), so\n\n\\[\\log |J_n(\\hat\\theta)|=k\\log n+O_p(1).\\]\n\nThe resulting expansion is\n\n\\[\\log p(y\\mid M) = \\hat\\ell-\\frac{k}{2}\\log n+O_p(1),\\]\n\nor, equivalently,\n\n\\[-2\\log p(y\\mid M) = \\mathrm{BIC}(M)+O_p(1).\\]\n\nThus BIC retains the terms that grow with sample size and discards terms that remain bounded in probability. These omitted terms include effects of the parameter prior and local likelihood curvature. BIC therefore approximates Bayesian evidence without explicitly specifying all prior details, but it is not an exact prior-free marginal likelihood.\n\nBayes factors and model probabilities\n\nFor two models \\(M_1\\) and \\(M_2\\), their Bayes factor is\n\n\\[BF_{12}=\\frac{p(y\\mid M_1)}{p(y\\mid M_2)}.\\]\n\nThe BIC approximation gives\n\n\\[2\\log BF_{12} \\approx \\mathrm{BIC}(M_2)-\\mathrm{BIC}(M_1).\\]\n\nA lower BIC for \\(M_1\\) therefore corresponds to approximate evidence in its favor.\n\nUsing Bayes’ theorem, approximate model probabilities can be constructed as\n\n\\[P(M_i\\mid y) \\approx \\frac{\\exp(-\\Delta_i/2)P(M_i)} {\\sum_j\\exp(-\\Delta_j/2)P(M_j)},\\]\n\nwhere\n\n\\[\\Delta_i=\\mathrm{BIC}(M_i)-\\min_j\\mathrm{BIC}(M_j).\\]\n\nWith equal prior probabilities for the models, the prior factors cancel. These probabilities are conditional on the candidate set and on the accuracy of the BIC approximation; they are not exact posterior probabilities."
    },
    {
      "heading": "Comparing models",
      "text": "For two models, define\n\n\\[\\Delta\\mathrm{BIC} = \\mathrm{BIC}(M_2)-\\mathrm{BIC}(M_1).\\]\n\nPositive values favor \\(M_1\\), while negative values favor \\(M_2\\). Informal tables sometimes classify differences as weak, moderate, or strong evidence. Such labels are conventions, not universal decision thresholds or p-values.\n\nSuppose two models are fitted to \\(n=100\\) observations:\n\nModel | Parameters \\(k\\) | Maximized log-likelihood \\(\\hat\\ell\\) | BIC\n\\(M_1\\) | 3 | \\(-140\\) | \\(293.82\\)\n\\(M_2\\) | 5 | \\(-136\\) | \\(295.03\\)\n\nThe second model improves twice the log-likelihood by \\(8\\), but its two additional parameters incur a penalty of\n\n\\[2\\log 100\\approx9.21.\\]\n\nBIC therefore favors \\(M_1\\), although the difference of approximately \\(1.21\\) is small.\n\nMore generally, adding \\(q\\) parameters is favored only if\n\n\\[2(\\hat\\ell_{\\mathrm{larger}}-\\hat\\ell_{\\mathrm{smaller}}) > q\\log n.\\]\n\nUnlike a conventional fixed-level likelihood-ratio test in hypothesis testing, this comparison uses a threshold that increases with sample size."
    },
    {
      "heading": "Gaussian linear regression",
      "text": "For linear regression with independent errors following a normal distribution of common unknown variance, let \\(p\\) be the number of regression coefficients, including any intercept, and let\n\n\\[\\mathrm{RSS}=\\sum_{i=1}^{n}(y_i-\\hat y_i)^2.\\]\n\nAt the maximum likelihood estimate of the error variance,\n\n\\[\\hat\\sigma^2=\\frac{\\mathrm{RSS}}{n}.\\]\n\nThe maximized likelihood yields\n\n\\[-2\\hat\\ell = n\\log\\!\\left(\\frac{\\mathrm{RSS}}{n}\\right) +n\\bigl[1+\\log(2\\pi)\\bigr].\\]\n\nCounting the variance parameter gives \\(k=p+1\\), so\n\n\\[\\mathrm{BIC} = n\\log\\!\\left(\\frac{\\mathrm{RSS}}{n}\\right) +(p+1)\\log n +n\\bigl[1+\\log(2\\pi)\\bigr].\\]\n\nFor models fitted to the same response observations, the final term is common and can be dropped. The common penalty for the variance parameter can also be omitted without changing the ranking, producing the frequently used comparison formula\n\n\\[n\\log(\\mathrm{RSS}/n)+p\\log n.\\]\n\nThis simplified expression is specific to the stated Gaussian likelihood. Applying it unchanged to other response distributions, correlated errors, or differently defined residual sums of squares does not generally produce the corresponding BIC."
    },
    {
      "heading": "Large-sample properties",
      "text": "An important property of BIC is model-selection consistency. Under suitable regularity conditions, if a fixed finite candidate set contains the true model and its smallest correct representation is identifiable, the probability that BIC selects that representation approaches one as \\(n\\) increases.\n\nThe intuition differs for underfitting and overfitting:\n\n- An underfitted model typically loses log-likelihood at a rate proportional to \\(n\\), overwhelming the logarithmic complexity penalty.\n- A regular overfitted model typically gains only a bounded-in-probability amount of log-likelihood over the true smaller model, while its extra penalty grows as \\(\\log n\\).\n\nThis statement concerns repeated sampling and an asymptotic regime. It does not guarantee selection of the correct model in a finite dataset, nor does it establish that the candidate set contains the data-generating process.\n\nWhen all candidate models are misspecified, BIC may favor a model with better expected log-likelihood, related to its Kullback–Leibler divergence from the data-generating distribution. However, its classical consistency result does not automatically establish optimal prediction under misspecification or when model dimensions grow with sample size."
    },
    {
      "heading": "Comparison with AIC and cross-validation",
      "text": "The Akaike information criterion is\n\n\\[\\mathrm{AIC}=-2\\hat\\ell+2k.\\]\n\nAIC is motivated by estimating expected out-of-sample information loss, whereas BIC is motivated by approximating marginal likelihood.\n\nFor \\(n>e^2\\), approximately \\(7.39\\), BIC assigns a larger penalty per parameter than AIC. Consequently, it often selects smaller models, although the actual choice depends on the likelihood improvements among candidates.\n\nTheir theoretical objectives differ:\n\n- BIC emphasizes identifying a finite-dimensional model under assumptions supporting model-selection consistency.\n- AIC emphasizes predictive performance and need not consistently select the smallest true model.\n- Cross-validation estimates performance on held-out observations according to a chosen loss or scoring rule.\n\nNeither criterion is uniformly superior. Consistent identification and predictive accuracy are different goals: a variable too weak to be selected reliably may still contribute to prediction, while an additional variable may improve training fit without improving out-of-sample performance."
    },
    {
      "heading": "Applications and limitations",
      "text": "BIC is used in regression variable selection, generalized linear models, time-series analysis, and selection of dependence structures or latent model components. Its simple formula makes it useful when many candidate likelihood-based models must be compared.\n\nSeveral qualifications govern its interpretation.\n\nComparable data and likelihoods\n\nCandidate models should describe the same observed data and use likelihoods on a compatible scale. BIC values from different response datasets, different subsets of observations, or likelihoods with inconsistently omitted constants are generally not directly comparable. Transforming the response can also require a Jacobian adjustment if likelihoods are to be compared on the same observation scale.\n\nDependence and the definition of sample size\n\nThe usual \\(k\\log n\\) penalty arises when information accumulates proportionally to \\(n\\). With clustered, longitudinal, spatial, or other dependent data, different parameters may accumulate information at different rates. Replacing \\(n\\) by an informal “effective sample size” is not a universally valid correction; an appropriate approximation depends on the likelihood and sampling structure.\n\nNonregular and singular models\n\nThe standard derivation assumes identifiability, an interior likelihood maximum, and nonsingular local information. These assumptions can fail for boundary parameters, latent-variable models, and Gaussian mixture models. In singular models, the leading complexity correction need not be \\(k\\log n\\). Standard BIC is nevertheless often used in such settings, but its usual evidence approximation and consistency arguments require separate justification.\n\nHigh-dimensional model searches\n\nClassical BIC assumes a fixed-dimensional asymptotic setting. When the number of parameters or candidate models grows rapidly with sample size, its penalty may not sufficiently account for the size of the search. Extensions such as the extended Bayesian information criterion introduce additional penalties under particular assumptions.\n\nFinite samples, priors, and estimation\n\nIn small samples, discarded terms in the evidence approximation can materially affect model rankings. Marginal likelihood can also depend strongly on prior scale, a dependence ordinary BIC suppresses rather than eliminates. Moreover, the formula presumes a maximized likelihood: substituting a local optimum, a penalized fit, or a variational bound changes its interpretation.\n\nFinally, choosing the smallest BIC does not demonstrate good absolute fit or account fully for uncertainty introduced by model selection. A selected model can still have inappropriate distributional assumptions, misspecified dependence, or poor predictive calibration."
    }
  ],
  "links": [
    "maximum-likelihood-estimation",
    "likelihood-function",
    "natural-logarithm",
    "regularization",
    "bayesian-inference",
    "marginal-likelihood",
    "prior-distribution",
    "laplace-approximation",
    "hessian-matrix",
    "bayes-factor",
    "bayes-theorem",
    "p-value",
    "statistical-hypothesis-testing",
    "linear-regression",
    "normal-distribution",
    "kullback-leibler-divergence",
    "akaike-information-criterion",
    "cross-validation",
    "generalized-linear-model",
    "time-series",
    "statistical-identification",
    "gaussian-mixture-model"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}