The Akaike information criterion (AIC) is a criterion in statistics for comparing models fitted to the same observations. It combines a model’s maximized likelihood with a penalty for its number of estimated parameters, favoring models expected to predict new observations well rather than merely fit the observed data closely. For a model with freely estimated parameters and maximized likelihood ,
Among comparable candidate models, a smaller AIC indicates a more favorable estimated trade-off between fit and complexity. AIC is a relative criterion: its absolute value does not establish whether a model is adequate, and the model with the lowest AIC need not describe the data-generating process correctly.
Definition and interpretation
Suppose a model assigns a probability density or mass function to the observed data , where contains freely estimated parameters. Let be the maximum-likelihood estimate, and define the maximized log likelihood as
Then
The two terms serve different purposes:
- Fit: decreases as the maximized likelihood increases.
- Complexity correction: compensates, approximately, for the optimism introduced by estimating parameters from the same observations used to evaluate fit.
Adding parameters cannot reduce the maximized likelihood when the simpler model is nested within the larger one. Consequently, likelihood alone tends to favor increasingly flexible models. AIC requires an improvement in log likelihood large enough to offset the additional parameter penalty.
The logarithm is the natural logarithm. AIC can be positive or negative because likelihood values, particularly likelihoods formed from continuous densities, are not restricted to the interval from zero to one. Only differences calculated on a common likelihood basis have the intended interpretation.
Information-theoretic foundation
AIC is motivated by information theory, particularly Kullback–Leibler divergence. If the unknown data-generating density is , the divergence from to a fitted model is
This can be written as
The first term is the same for every candidate model. Thus, minimizing expected divergence is equivalent to maximizing the expected log likelihood of new observations. Equivalently, AIC targets predictive cross-entropy, up to terms common to all candidates and an overall scale.
The difficulty is that the observed maximized log likelihood is too favorable an assessment of prediction: the parameters were chosen precisely to fit those observations. Under standard large-sample regularity conditions, with a correctly specified model, the expected difference between the training log likelihood and the log likelihood of an independent dataset of the same size is approximately . On the negative twice-log-likelihood scale, the correction is therefore .
AIC consequently estimates expected out-of-sample information loss up to an unknown additive constant. It does not directly estimate an absolute Kullback–Leibler divergence.
Assumptions behind the correction
The familiar penalty relies on regular likelihood behavior. Typical assumptions include identifiable parameters, a fixed finite parameter dimension, sufficiently smooth likelihoods, and a nonsingular Fisher information matrix, with the relevant parameter value away from problematic boundaries.
The standard derivation also uses correct model specification. Under misspecification, the general asymptotic optimism correction involves both the expected curvature of the log likelihood and the variability of its score. These quantities need not coincide, so is not a universally valid correction.
This distinction matters because AIC is often used when all candidate models are understood as approximations. Its information-theoretic motivation remains useful, but the accuracy of its numerical penalty depends on how well the regular approximation applies.
Comparing candidate models
For candidate models , define
The best-ranked model has . Larger differences indicate less support relative to the best-ranked candidate under the criterion.
A common summary is the Akaike weight,
These weights sum to one and express relative support within the specified candidate set. They depend on which models are included and are not automatically posterior probabilities that the models are true.
Differences of roughly two AIC units are often treated as indicating little separation, while substantially larger differences suggest clearer separation. Such thresholds are informal conventions, not universal decision rules or significance levels.
Numerical example
Consider three models fitted to the same observations:
| Model | Estimated parameters | Maximized log likelihood | AIC | |
|---|---|---|---|---|
| A | 3 | 246 | 2 | |
| B | 5 | 244 | 0 | |
| C | 8 | 248 | 4 |
Model C fits the observed data best in likelihood terms, but its additional parameters do not produce enough improvement to offset the AIC penalty. Model B has the lowest AIC.
The corresponding weights are approximately , , and . These quantify relative support among these three candidates; they do not establish that model B is an adequate description of the observations.
Parameter counting and likelihood conventions
The parameter count includes all freely estimated parameters in the likelihood, not merely the coefficients of primary interest. Depending on the model, this may include intercepts, error variances, dispersion parameters, and dependence parameters.
For example, in linear regression with independent normal errors,
a model with regression coefficients and an unknown error variance ordinarily has
If an intercept is estimated, it is included among the coefficients.
At the maximum-likelihood estimates,
where is the residual sum of squares. Thus, apart from a constant common to models fitted to the same response observations,
The variance estimate used here is , the maximum-likelihood estimate, rather than the usual unbiased estimate .
For penalized fitting methods such as ridge regression, the raw number of coefficients generally does not measure effective flexibility appropriately. Extensions may use effective degrees of freedom, but ordinary AIC cannot simply be transferred unchanged to every penalized estimator.
Small-sample correction
When the sample size is not large relative to the number of estimated parameters, the ordinary AIC correction can be insufficient. A widely used corrected form is
where is the sample size and includes all estimated likelihood parameters under this convention.
This expression is associated with particular model settings, notably Gaussian linear regression, and is often used as an approximation more broadly. Finite-sample corrections are model-dependent; the same expression is not an exact correction for every likelihood.
The additional penalty increases rapidly as approaches . For fixed , it tends to zero as increases. The displayed expression requires ; values outside that range do not provide a valid small-sample comparison.
Conditions for meaningful comparisons
AIC comparisons require likelihoods that represent comparable predictive tasks.
The observations must be comparable. Models fitted to different subsets of observations generally cannot be compared using their unadjusted AIC values. For example, different missing-data patterns can cause regression models to use different sample sizes.
The response and its probability scale must be handled consistently. Comparing a model for with a model for requires the appropriate change-of-variable adjustment so that both likelihoods describe the same observations on a common scale.
Likelihood constants must be treated consistently. Terms independent of the parameters may be irrelevant for estimating a single model but necessary when comparing likelihoods across model families or software implementations.
The fitting method matters. Ordinary AIC is based on maximized likelihood, not arbitrary goodness-of-fit scores. Restricted maximum likelihood values generally cannot be compared directly across models with different fixed-effect structures.
The dependence structure must be represented. For time series or clustered observations, treating dependent measurements as independent changes the likelihood and can invalidate the intended comparison. Conditional and full likelihoods may also use different information and require consistent treatment.
Candidate models need not be nested. AIC can compare different model families when their likelihoods describe the same observed outcome on a compatible basis.
Relationship to other methods
Cross-validation
Both AIC and cross-validation address predictive performance rather than training fit alone. In regular parametric settings, AIC and leave-one-out cross-validation have closely related large-sample behavior for log-likelihood prediction.
Their equivalence is not universal. Cross-validation can evaluate other loss functions, and its results depend on how observations are divided. For dependent observations, validation schemes must reflect the intended prediction task.
Bayesian information criterion
The Bayesian information criterion is
Unlike AIC’s fixed penalty of two per parameter, BIC’s penalty increases with sample size. BIC is associated with a large-sample approximation to Bayesian model evidence under particular assumptions.
The criteria pursue different objectives. Under suitable conditions and when the true model is among a fixed collection of candidates, BIC can consistently select it as sample size grows. AIC generally is not model-selection consistent in this sense: it can retain a positive probability of selecting an unnecessarily large model. This behavior does not contradict its predictive objective, which differs from identifying a single true finite-dimensional model.
Hypothesis testing
AIC is not a procedure in statistical hypothesis testing. It supplies neither a p-value nor a fixed type-I error rate. Although an AIC comparison of nested models can be expressed using their likelihood-ratio statistic, its penalty is not chosen to implement a conventional significance threshold.
Uses and limitations
AIC is used in model selection for regression, generalized linear models, autoregressive modeling, ecological modeling, and many other likelihood-based applications. It is particularly useful when several scientifically plausible models represent different approximations to a process.
Its main limitations include:
- Relative rather than absolute evaluation. AIC ranks the supplied candidates even if all fit poorly.
- Dependence on the candidate set. Omitted models cannot receive support, and adding candidates changes Akaike weights.
- Approximation error. Small samples, weak identification, boundary parameters, and irregular likelihoods can undermine the standard penalty.
- Extensive model searching. Searching very large model collections can introduce selection optimism beyond the correction for estimating parameters within each model.
- Objective mismatch. A model favored for log-density prediction need not be best for classification accuracy, causal estimation, or a particular forecasting loss.
- Post-selection uncertainty. Standard parameter intervals calculated as though the selected model had been fixed in advance generally omit uncertainty introduced by model selection.
Selecting a model by AIC does not establish causal relationships, eliminate confounding, or validate its substantive assumptions. Its role is narrower: estimating relative predictive information loss under a specified likelihood framework.
Historical development
Hirotugu Akaike introduced the information-theoretic approach in work presented in the early 1970s, including a 1973 publication on information theory and maximum likelihood. His 1974 paper, A New Look at the Statistical Model Identification, helped establish the criterion widely.
The abbreviation AIC was used for “An Information Criterion” and subsequently became widely known as the Akaike information criterion. Later work developed finite-sample corrections, extensions for misspecified models, and related methods for evaluating more complex predictive models.