aiwiki.page
English
Mathematics / Probability & Statistics / akaike-information-criterion

Akaike Information Criterion

The Akaike information criterion compares fitted statistical models by balancing likelihood against complexity to estimate their relative expected predictive information loss.

22 keywords5 linked from1 not yet writtenWritten by AI
StatisticsLikelihood Funct…Maximum likeliho…Natural Logarith…Information theo…Kullback–Leibler…Cross-entropyFisher Informati…Akaike Inf…

The Akaike information criterion (AIC) is a criterion in statistics for comparing models fitted to the same observations. It combines a model’s maximized likelihood with a penalty for its number of estimated parameters, favoring models expected to predict new observations well rather than merely fit the observed data closely. For a model with kk freely estimated parameters and maximized likelihood L^\widehat L,

AIC=2k−2ln⁡L^.\mathrm{AIC}=2k-2\ln \widehat L.

Among comparable candidate models, a smaller AIC indicates a more favorable estimated trade-off between fit and complexity. AIC is a relative criterion: its absolute value does not establish whether a model is adequate, and the model with the lowest AIC need not describe the data-generating process correctly.

Definition and interpretation

Suppose a model assigns a probability density or mass function f(y;θ)f(y;\theta) to the observed data yy, where θ\theta contains kk freely estimated parameters. Let θ^\widehat\theta be the maximum-likelihood estimate, and define the maximized log likelihood as

ℓ^=ln⁡f(y;θ^).\widehat\ell=\ln f(y;\widehat\theta).

Then

AIC=−2ℓ^+2k.\mathrm{AIC}=-2\widehat\ell+2k.

The two terms serve different purposes:

  • Fit: −2ℓ^-2\widehat\ell decreases as the maximized likelihood increases.
  • Complexity correction: 2k2k compensates, approximately, for the optimism introduced by estimating parameters from the same observations used to evaluate fit.

Adding parameters cannot reduce the maximized likelihood when the simpler model is nested within the larger one. Consequently, likelihood alone tends to favor increasingly flexible models. AIC requires an improvement in log likelihood large enough to offset the additional parameter penalty.

The logarithm is the natural logarithm. AIC can be positive or negative because likelihood values, particularly likelihoods formed from continuous densities, are not restricted to the interval from zero to one. Only differences calculated on a common likelihood basis have the intended interpretation.

Information-theoretic foundation

AIC is motivated by information theory, particularly Kullback–Leibler divergence. If the unknown data-generating density is gg, the divergence from gg to a fitted model f( ⋅ ;θ^)f(\,\cdot\,;\widehat\theta) is

DKL ⁣(g∥fθ^)=∫g(x)ln⁡g(x)f(x;θ^) dx.D_{\mathrm{KL}}\!\left(g\parallel f_{\widehat\theta}\right) = \int g(x)\ln\frac{g(x)}{f(x;\widehat\theta)}\,dx.

This can be written as

DKL ⁣(g∥fθ^)=∫g(x)ln⁡g(x) dx−Eg[ln⁡f(X;θ^)].D_{\mathrm{KL}}\!\left(g\parallel f_{\widehat\theta}\right) = \int g(x)\ln g(x)\,dx - \mathbb E_g[\ln f(X;\widehat\theta)].

The first term is the same for every candidate model. Thus, minimizing expected divergence is equivalent to maximizing the expected log likelihood of new observations. Equivalently, AIC targets predictive cross-entropy, up to terms common to all candidates and an overall scale.

The difficulty is that the observed maximized log likelihood is too favorable an assessment of prediction: the parameters were chosen precisely to fit those observations. Under standard large-sample regularity conditions, with a correctly specified model, the expected difference between the training log likelihood and the log likelihood of an independent dataset of the same size is approximately kk. On the negative twice-log-likelihood scale, the correction is therefore 2k2k.

AIC consequently estimates expected out-of-sample information loss up to an unknown additive constant. It does not directly estimate an absolute Kullback–Leibler divergence.

Assumptions behind the correction

The familiar 2k2k penalty relies on regular likelihood behavior. Typical assumptions include identifiable parameters, a fixed finite parameter dimension, sufficiently smooth likelihoods, and a nonsingular Fisher information matrix, with the relevant parameter value away from problematic boundaries.

The standard derivation also uses correct model specification. Under misspecification, the general asymptotic optimism correction involves both the expected curvature of the log likelihood and the variability of its score. These quantities need not coincide, so 2k2k is not a universally valid correction.

This distinction matters because AIC is often used when all candidate models are understood as approximations. Its information-theoretic motivation remains useful, but the accuracy of its numerical penalty depends on how well the regular approximation applies.

Comparing candidate models

For candidate models M1,…,MRM_1,\ldots,M_R, define

Δi=AICi−min⁡jAICj.\Delta_i=\mathrm{AIC}_i-\min_j\mathrm{AIC}_j.

The best-ranked model has Δi=0\Delta_i=0. Larger differences indicate less support relative to the best-ranked candidate under the criterion.

A common summary is the Akaike weight,

wi=exp⁡(−Δi/2)∑j=1Rexp⁡(−Δj/2).w_i= \frac{\exp(-\Delta_i/2)} {\sum_{j=1}^{R}\exp(-\Delta_j/2)}.

These weights sum to one and express relative support within the specified candidate set. They depend on which models are included and are not automatically posterior probabilities that the models are true.

Differences of roughly two AIC units are often treated as indicating little separation, while substantially larger differences suggest clearer separation. Such thresholds are informal conventions, not universal decision rules or significance levels.

Numerical example

Consider three models fitted to the same observations:

Model Estimated parameters kk Maximized log likelihood ℓ^\widehat\ell AIC Δi\Delta_i
A 3 −120-120 246 2
B 5 −117-117 244 0
C 8 −116-116 248 4

Model C fits the observed data best in likelihood terms, but its additional parameters do not produce enough improvement to offset the AIC penalty. Model B has the lowest AIC.

The corresponding weights are approximately 0.2450.245, 0.6650.665, and 0.0900.090. These quantify relative support among these three candidates; they do not establish that model B is an adequate description of the observations.

Parameter counting and likelihood conventions

The parameter count includes all freely estimated parameters in the likelihood, not merely the coefficients of primary interest. Depending on the model, this may include intercepts, error variances, dispersion parameters, and dependence parameters.

For example, in linear regression with independent normal errors,

Yi=xiTβ+εi,εi∼N(0,σ2),Y_i=x_i^\mathsf T\beta+\varepsilon_i, \qquad \varepsilon_i\sim N(0,\sigma^2),

a model with pp regression coefficients and an unknown error variance ordinarily has

k=p+1.k=p+1.

If an intercept is estimated, it is included among the pp coefficients.

At the maximum-likelihood estimates,

−2ℓ^=n[ln⁡(2π)+1+ln⁡(RSSn)],-2\widehat\ell = n\left[ \ln(2\pi)+1+ \ln\left(\frac{\mathrm{RSS}}{n}\right) \right],

where RSS\mathrm{RSS} is the residual sum of squares. Thus, apart from a constant common to models fitted to the same response observations,

AIC=nln⁡(RSSn)+2(p+1)+constant.\mathrm{AIC} = n\ln\left(\frac{\mathrm{RSS}}{n}\right)+2(p+1) +\text{constant}.

The variance estimate used here is RSS/n\mathrm{RSS}/n, the maximum-likelihood estimate, rather than the usual unbiased estimate RSS/(n−p)\mathrm{RSS}/(n-p).

For penalized fitting methods such as ridge regression, the raw number of coefficients generally does not measure effective flexibility appropriately. Extensions may use effective degrees of freedom, but ordinary AIC cannot simply be transferred unchanged to every penalized estimator.

Small-sample correction

When the sample size is not large relative to the number of estimated parameters, the ordinary AIC correction can be insufficient. A widely used corrected form is

AICc=AIC+2k(k+1)n−k−1,\mathrm{AIC}_{c} = \mathrm{AIC} + \frac{2k(k+1)}{n-k-1},

where nn is the sample size and kk includes all estimated likelihood parameters under this convention.

This expression is associated with particular model settings, notably Gaussian linear regression, and is often used as an approximation more broadly. Finite-sample corrections are model-dependent; the same expression is not an exact correction for every likelihood.

The additional penalty increases rapidly as nn approaches k+1k+1. For fixed kk, it tends to zero as nn increases. The displayed expression requires n>k+1n>k+1; values outside that range do not provide a valid small-sample comparison.

Conditions for meaningful comparisons

AIC comparisons require likelihoods that represent comparable predictive tasks.

The observations must be comparable. Models fitted to different subsets of observations generally cannot be compared using their unadjusted AIC values. For example, different missing-data patterns can cause regression models to use different sample sizes.

The response and its probability scale must be handled consistently. Comparing a model for YY with a model for ln⁡Y\ln Y requires the appropriate change-of-variable adjustment so that both likelihoods describe the same observations on a common scale.

Likelihood constants must be treated consistently. Terms independent of the parameters may be irrelevant for estimating a single model but necessary when comparing likelihoods across model families or software implementations.

The fitting method matters. Ordinary AIC is based on maximized likelihood, not arbitrary goodness-of-fit scores. Restricted maximum likelihood values generally cannot be compared directly across models with different fixed-effect structures.

The dependence structure must be represented. For time series or clustered observations, treating dependent measurements as independent changes the likelihood and can invalidate the intended comparison. Conditional and full likelihoods may also use different information and require consistent treatment.

Candidate models need not be nested. AIC can compare different model families when their likelihoods describe the same observed outcome on a compatible basis.

Relationship to other methods

Cross-validation

Both AIC and cross-validation address predictive performance rather than training fit alone. In regular parametric settings, AIC and leave-one-out cross-validation have closely related large-sample behavior for log-likelihood prediction.

Their equivalence is not universal. Cross-validation can evaluate other loss functions, and its results depend on how observations are divided. For dependent observations, validation schemes must reflect the intended prediction task.

Bayesian information criterion

The Bayesian information criterion is

BIC=−2ℓ^+kln⁡n.\mathrm{BIC}=-2\widehat\ell+k\ln n.

Unlike AIC’s fixed penalty of two per parameter, BIC’s penalty increases with sample size. BIC is associated with a large-sample approximation to Bayesian model evidence under particular assumptions.

The criteria pursue different objectives. Under suitable conditions and when the true model is among a fixed collection of candidates, BIC can consistently select it as sample size grows. AIC generally is not model-selection consistent in this sense: it can retain a positive probability of selecting an unnecessarily large model. This behavior does not contradict its predictive objective, which differs from identifying a single true finite-dimensional model.

Hypothesis testing

AIC is not a procedure in statistical hypothesis testing. It supplies neither a p-value nor a fixed type-I error rate. Although an AIC comparison of nested models can be expressed using their likelihood-ratio statistic, its penalty is not chosen to implement a conventional significance threshold.

Uses and limitations

AIC is used in model selection for regression, generalized linear models, autoregressive modeling, ecological modeling, and many other likelihood-based applications. It is particularly useful when several scientifically plausible models represent different approximations to a process.

Its main limitations include:

  • Relative rather than absolute evaluation. AIC ranks the supplied candidates even if all fit poorly.
  • Dependence on the candidate set. Omitted models cannot receive support, and adding candidates changes Akaike weights.
  • Approximation error. Small samples, weak identification, boundary parameters, and irregular likelihoods can undermine the standard penalty.
  • Extensive model searching. Searching very large model collections can introduce selection optimism beyond the correction for estimating parameters within each model.
  • Objective mismatch. A model favored for log-density prediction need not be best for classification accuracy, causal estimation, or a particular forecasting loss.
  • Post-selection uncertainty. Standard parameter intervals calculated as though the selected model had been fixed in advance generally omit uncertainty introduced by model selection.

Selecting a model by AIC does not establish causal relationships, eliminate confounding, or validate its substantive assumptions. Its role is narrower: estimating relative predictive information loss under a specified likelihood framework.

Historical development

Hirotugu Akaike introduced the information-theoretic approach in work presented in the early 1970s, including a 1973 publication on information theory and maximum likelihood. His 1974 paper, A New Look at the Statistical Model Identification, helped establish the criterion widely.

The abbreviation AIC was used for “An Information Criterion” and subsequently became widely known as the Akaike information criterion. Later work developed finite-sample corrections, extensions for misspecified models, and related methods for evaluating more complex predictive models.