{
  "id": "akaike-information-criterion",
  "title": "Akaike Information Criterion",
  "lang": "en",
  "source_lang": "en",
  "summary": "The Akaike information criterion compares fitted statistical models by balancing likelihood against complexity to estimate their relative expected predictive information loss.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "AIC",
    "Akaike’s information criterion",
    "An Information Criterion"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/akaike-information-criterion/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/akaike-information-criterion/",
      "json": "https://aiwiki.page/zh/akaike-information-criterion.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The Akaike information criterion (AIC) is a criterion in statistics for comparing models fitted to the same observations. It combines a model’s maximized likelihood with a penalty for its number of estimated parameters, favoring models expected to predict new observations well rather than merely fit the observed data closely. For a model with \\(k\\) freely estimated parameters and maximized likelihood \\(\\widehat L\\),\n\n\\[\\mathrm{AIC}=2k-2\\ln \\widehat L.\\]\n\nAmong comparable candidate models, a smaller AIC indicates a more favorable estimated trade-off between fit and complexity. AIC is a relative criterion: its absolute value does not establish whether a model is adequate, and the model with the lowest AIC need not describe the data-generating process correctly."
    },
    {
      "heading": "Definition and interpretation",
      "text": "Suppose a model assigns a probability density or mass function \\(f(y;\\theta)\\) to the observed data \\(y\\), where \\(\\theta\\) contains \\(k\\) freely estimated parameters. Let \\(\\widehat\\theta\\) be the maximum-likelihood estimate, and define the maximized log likelihood as\n\n\\[\\widehat\\ell=\\ln f(y;\\widehat\\theta).\\]\n\nThen\n\n\\[\\mathrm{AIC}=-2\\widehat\\ell+2k.\\]\n\nThe two terms serve different purposes:\n\n- Fit: \\(-2\\widehat\\ell\\) decreases as the maximized likelihood increases.\n- Complexity correction: \\(2k\\) compensates, approximately, for the optimism introduced by estimating parameters from the same observations used to evaluate fit.\n\nAdding parameters cannot reduce the maximized likelihood when the simpler model is nested within the larger one. Consequently, likelihood alone tends to favor increasingly flexible models. AIC requires an improvement in log likelihood large enough to offset the additional parameter penalty.\n\nThe logarithm is the natural logarithm. AIC can be positive or negative because likelihood values, particularly likelihoods formed from continuous densities, are not restricted to the interval from zero to one. Only differences calculated on a common likelihood basis have the intended interpretation."
    },
    {
      "heading": "Information-theoretic foundation",
      "text": "AIC is motivated by information theory, particularly Kullback–Leibler divergence. If the unknown data-generating density is \\(g\\), the divergence from \\(g\\) to a fitted model \\(f(\\,\\cdot\\,;\\widehat\\theta)\\) is\n\n\\[D_{\\mathrm{KL}}\\!\\left(g\\parallel f_{\\widehat\\theta}\\right) = \\int g(x)\\ln\\frac{g(x)}{f(x;\\widehat\\theta)}\\,dx.\\]\n\nThis can be written as\n\n\\[D_{\\mathrm{KL}}\\!\\left(g\\parallel f_{\\widehat\\theta}\\right) = \\int g(x)\\ln g(x)\\,dx - \\mathbb E_g[\\ln f(X;\\widehat\\theta)].\\]\n\nThe first term is the same for every candidate model. Thus, minimizing expected divergence is equivalent to maximizing the expected log likelihood of new observations. Equivalently, AIC targets predictive cross-entropy, up to terms common to all candidates and an overall scale.\n\nThe difficulty is that the observed maximized log likelihood is too favorable an assessment of prediction: the parameters were chosen precisely to fit those observations. Under standard large-sample regularity conditions, with a correctly specified model, the expected difference between the training log likelihood and the log likelihood of an independent dataset of the same size is approximately \\(k\\). On the negative twice-log-likelihood scale, the correction is therefore \\(2k\\).\n\nAIC consequently estimates expected out-of-sample information loss up to an unknown additive constant. It does not directly estimate an absolute Kullback–Leibler divergence.\n\nAssumptions behind the correction\n\nThe familiar \\(2k\\) penalty relies on regular likelihood behavior. Typical assumptions include identifiable parameters, a fixed finite parameter dimension, sufficiently smooth likelihoods, and a nonsingular Fisher information matrix, with the relevant parameter value away from problematic boundaries.\n\nThe standard derivation also uses correct model specification. Under misspecification, the general asymptotic optimism correction involves both the expected curvature of the log likelihood and the variability of its score. These quantities need not coincide, so \\(2k\\) is not a universally valid correction.\n\nThis distinction matters because AIC is often used when all candidate models are understood as approximations. Its information-theoretic motivation remains useful, but the accuracy of its numerical penalty depends on how well the regular approximation applies."
    },
    {
      "heading": "Comparing candidate models",
      "text": "For candidate models \\(M_1,\\ldots,M_R\\), define\n\n\\[\\Delta_i=\\mathrm{AIC}_i-\\min_j\\mathrm{AIC}_j.\\]\n\nThe best-ranked model has \\(\\Delta_i=0\\). Larger differences indicate less support relative to the best-ranked candidate under the criterion.\n\nA common summary is the Akaike weight,\n\n\\[w_i= \\frac{\\exp(-\\Delta_i/2)} {\\sum_{j=1}^{R}\\exp(-\\Delta_j/2)}.\\]\n\nThese weights sum to one and express relative support within the specified candidate set. They depend on which models are included and are not automatically posterior probabilities that the models are true.\n\nDifferences of roughly two AIC units are often treated as indicating little separation, while substantially larger differences suggest clearer separation. Such thresholds are informal conventions, not universal decision rules or significance levels.\n\nNumerical example\n\nConsider three models fitted to the same observations:\n\nModel | Estimated parameters \\(k\\) | Maximized log likelihood \\(\\widehat\\ell\\) | AIC | \\(\\Delta_i\\)\nA | 3 | \\(-120\\) | 246 | 2\nB | 5 | \\(-117\\) | 244 | 0\nC | 8 | \\(-116\\) | 248 | 4\n\nModel C fits the observed data best in likelihood terms, but its additional parameters do not produce enough improvement to offset the AIC penalty. Model B has the lowest AIC.\n\nThe corresponding weights are approximately \\(0.245\\), \\(0.665\\), and \\(0.090\\). These quantify relative support among these three candidates; they do not establish that model B is an adequate description of the observations."
    },
    {
      "heading": "Parameter counting and likelihood conventions",
      "text": "The parameter count includes all freely estimated parameters in the likelihood, not merely the coefficients of primary interest. Depending on the model, this may include intercepts, error variances, dispersion parameters, and dependence parameters.\n\nFor example, in linear regression with independent normal errors,\n\n\\[Y_i=x_i^\\mathsf T\\beta+\\varepsilon_i, \\qquad \\varepsilon_i\\sim N(0,\\sigma^2),\\]\n\na model with \\(p\\) regression coefficients and an unknown error variance ordinarily has\n\n\\[k=p+1.\\]\n\nIf an intercept is estimated, it is included among the \\(p\\) coefficients.\n\nAt the maximum-likelihood estimates,\n\n\\[-2\\widehat\\ell = n\\left[ \\ln(2\\pi)+1+ \\ln\\left(\\frac{\\mathrm{RSS}}{n}\\right) \\right],\\]\n\nwhere \\(\\mathrm{RSS}\\) is the residual sum of squares. Thus, apart from a constant common to models fitted to the same response observations,\n\n\\[\\mathrm{AIC} = n\\ln\\left(\\frac{\\mathrm{RSS}}{n}\\right)+2(p+1) +\\text{constant}.\\]\n\nThe variance estimate used here is \\(\\mathrm{RSS}/n\\), the maximum-likelihood estimate, rather than the usual unbiased estimate \\(\\mathrm{RSS}/(n-p)\\).\n\nFor penalized fitting methods such as ridge regression, the raw number of coefficients generally does not measure effective flexibility appropriately. Extensions may use effective degrees of freedom, but ordinary AIC cannot simply be transferred unchanged to every penalized estimator."
    },
    {
      "heading": "Small-sample correction",
      "text": "When the sample size is not large relative to the number of estimated parameters, the ordinary AIC correction can be insufficient. A widely used corrected form is\n\n\\[\\mathrm{AIC}_{c} = \\mathrm{AIC} + \\frac{2k(k+1)}{n-k-1},\\]\n\nwhere \\(n\\) is the sample size and \\(k\\) includes all estimated likelihood parameters under this convention.\n\nThis expression is associated with particular model settings, notably Gaussian linear regression, and is often used as an approximation more broadly. Finite-sample corrections are model-dependent; the same expression is not an exact correction for every likelihood.\n\nThe additional penalty increases rapidly as \\(n\\) approaches \\(k+1\\). For fixed \\(k\\), it tends to zero as \\(n\\) increases. The displayed expression requires \\(n>k+1\\); values outside that range do not provide a valid small-sample comparison."
    },
    {
      "heading": "Conditions for meaningful comparisons",
      "text": "AIC comparisons require likelihoods that represent comparable predictive tasks.\n\nThe observations must be comparable. Models fitted to different subsets of observations generally cannot be compared using their unadjusted AIC values. For example, different missing-data patterns can cause regression models to use different sample sizes.\n\nThe response and its probability scale must be handled consistently. Comparing a model for \\(Y\\) with a model for \\(\\ln Y\\) requires the appropriate change-of-variable adjustment so that both likelihoods describe the same observations on a common scale.\n\nLikelihood constants must be treated consistently. Terms independent of the parameters may be irrelevant for estimating a single model but necessary when comparing likelihoods across model families or software implementations.\n\nThe fitting method matters. Ordinary AIC is based on maximized likelihood, not arbitrary goodness-of-fit scores. Restricted maximum likelihood values generally cannot be compared directly across models with different fixed-effect structures.\n\nThe dependence structure must be represented. For time series or clustered observations, treating dependent measurements as independent changes the likelihood and can invalidate the intended comparison. Conditional and full likelihoods may also use different information and require consistent treatment.\n\nCandidate models need not be nested. AIC can compare different model families when their likelihoods describe the same observed outcome on a compatible basis."
    },
    {
      "heading": "Relationship to other methods",
      "text": "Cross-validation\n\nBoth AIC and cross-validation address predictive performance rather than training fit alone. In regular parametric settings, AIC and leave-one-out cross-validation have closely related large-sample behavior for log-likelihood prediction.\n\nTheir equivalence is not universal. Cross-validation can evaluate other loss functions, and its results depend on how observations are divided. For dependent observations, validation schemes must reflect the intended prediction task.\n\nBayesian information criterion\n\nThe Bayesian information criterion is\n\n\\[\\mathrm{BIC}=-2\\widehat\\ell+k\\ln n.\\]\n\nUnlike AIC’s fixed penalty of two per parameter, BIC’s penalty increases with sample size. BIC is associated with a large-sample approximation to Bayesian model evidence under particular assumptions.\n\nThe criteria pursue different objectives. Under suitable conditions and when the true model is among a fixed collection of candidates, BIC can consistently select it as sample size grows. AIC generally is not model-selection consistent in this sense: it can retain a positive probability of selecting an unnecessarily large model. This behavior does not contradict its predictive objective, which differs from identifying a single true finite-dimensional model.\n\nHypothesis testing\n\nAIC is not a procedure in statistical hypothesis testing. It supplies neither a p-value nor a fixed type-I error rate. Although an AIC comparison of nested models can be expressed using their likelihood-ratio statistic, its penalty is not chosen to implement a conventional significance threshold."
    },
    {
      "heading": "Uses and limitations",
      "text": "AIC is used in model selection for regression, generalized linear models, autoregressive modeling, ecological modeling, and many other likelihood-based applications. It is particularly useful when several scientifically plausible models represent different approximations to a process.\n\nIts main limitations include:\n\n- Relative rather than absolute evaluation. AIC ranks the supplied candidates even if all fit poorly.\n- Dependence on the candidate set. Omitted models cannot receive support, and adding candidates changes Akaike weights.\n- Approximation error. Small samples, weak identification, boundary parameters, and irregular likelihoods can undermine the standard penalty.\n- Extensive model searching. Searching very large model collections can introduce selection optimism beyond the correction for estimating parameters within each model.\n- Objective mismatch. A model favored for log-density prediction need not be best for classification accuracy, causal estimation, or a particular forecasting loss.\n- Post-selection uncertainty. Standard parameter intervals calculated as though the selected model had been fixed in advance generally omit uncertainty introduced by model selection.\n\nSelecting a model by AIC does not establish causal relationships, eliminate confounding, or validate its substantive assumptions. Its role is narrower: estimating relative predictive information loss under a specified likelihood framework."
    },
    {
      "heading": "Historical development",
      "text": "Hirotugu Akaike introduced the information-theoretic approach in work presented in the early 1970s, including a 1973 publication on information theory and maximum likelihood. His 1974 paper, A New Look at the Statistical Model Identification, helped establish the criterion widely.\n\nThe abbreviation AIC was used for “An Information Criterion” and subsequently became widely known as the Akaike information criterion. Later work developed finite-sample corrections, extensions for misspecified models, and related methods for evaluating more complex predictive models."
    }
  ],
  "links": [
    "statistics",
    "likelihood-function",
    "maximum-likelihood-estimation",
    "natural-logarithm",
    "information-theory",
    "kullback-leibler-divergence",
    "cross-entropy",
    "fisher-information",
    "posterior-distribution",
    "linear-regression",
    "normal-distribution",
    "ridge-regression",
    "degrees-of-freedom",
    "time-series",
    "cross-validation",
    "loss-function",
    "bayesian-information-criterion",
    "statistical-hypothesis-testing",
    "p-value",
    "generalized-linear-model",
    "causal-inference"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}