Multicollinearity is a condition in statistics in which two or more predictors in a statistical model are exactly or approximately related by a linear equation. In linear regression, exact multicollinearity prevents the unique estimation of all regression coefficients, while approximate multicollinearity can make individual coefficient estimates imprecise and sensitive to small changes in the data. It concerns relationships among predictors, not their relationships with the outcome, and does not necessarily prevent accurate prediction.
Mathematical definition
Consider the linear regression model
where is an -dimensional outcome vector, is an design matrix, contains the regression coefficients, and contains the errors. The columns of represent predictors and, when included, an intercept.
Exact multicollinearity occurs when there is a nonzero vector such that
The columns of then fail to satisfy linear independence, and its rank is less than . Equivalently, at least one column is a linear combination of the others.
Approximate multicollinearity, often called near multicollinearity, occurs when some nonzero produces an that is small relative to the scales of and . The matrix can retain full column rank while containing little information about certain combinations of coefficients.
The distinction is important: exact dependence is an algebraic property, whereas the practical severity of approximate dependence depends on predictor scaling, sample size, error variability, and the quantities being estimated.
Why pairwise correlations are insufficient
A large absolute correlation between two predictors can indicate multicollinearity, but it is not necessary. For example, suppose and are independent, centered predictors with equal variance, and
The three columns are exactly dependent even though neither correlation involving equals in absolute value.
More generally, a predictor can be closely approximated by a combination of many others without being strongly correlated with any single one. A table of pairwise correlations therefore cannot rule out multicollinearity.
Consequences for coefficient estimation
Exact dependence and non-identification
When has full column rank, the ordinary least squares estimator is
Under exact multicollinearity, has no inverse, so this expression is undefined.
The least-squares problem still has solutions, but the coefficient vector is not unique. If , then
for every real number . Thus, multiple coefficient vectors give identical fitted values. The difficulty is one of statistical identification: the observed design does not distinguish among those coefficient vectors.
Although the coefficients are not unique, the least-squares fitted vector is unique because it is the orthogonal projection of onto the column space of . Some coefficient combinations can also remain identifiable.
For example, if , then
The data can identify , but not and separately.
Software may handle exact dependence by omitting columns, imposing constraints, or selecting a solution through a generalized inverse. Such conventions do not supply information about otherwise unidentified coefficients.
Approximate dependence and uncertainty
Suppose has full column rank and the errors satisfy
The conditional covariance matrix of the ordinary least squares estimator is
Near dependence makes some directions in coefficient space weakly determined. The resulting estimates can have large standard errors and wide confidence intervals. Coefficient magnitudes or signs may change substantially when observations are added, removed, or slightly perturbed.
Multicollinearity alone does not create bias in ordinary least squares under the stated zero-conditional-mean assumption. It affects precision rather than automatically shifting estimates systematically away from their true values. It also does not by itself imply endogeneity, which concerns relationships between predictors and the error term.
A model can consequently show strong evidence that a group of predictors matters jointly while individual coefficient tests provide little evidence about their separate contributions.
Variance inflation factors
A common diagnostic is the variance inflation factor (VIF). In an ordinary least squares model with an intercept, regress predictor on all the other predictors, including an intercept. If the auxiliary regression has coefficient of determination , then
Under the classical constant-error-variance assumptions,
This separates three influences on precision:
- the error variance ;
- the observed spread of predictor ;
- the extent to which the remaining predictors explain .
A VIF of indicates that the centered predictor is orthogonal to the other centered predictors. A VIF of means that, holding error variance and the predictor’s centered sum of squares fixed, its coefficient variance is five times the corresponding orthogonal-design value. Its standard error is multiplied by , not by .
Thresholds such as or are commonly used as screening conventions, but they are not universal decision rules. A high VIF need not make an estimate unusable if the dataset contains sufficient information, while a low VIF does not guarantee a narrow confidence interval. The reciprocal,
is sometimes called tolerance.
Geometric and numerical interpretation
The singular value decomposition provides a useful description:
Exact multicollinearity corresponds to at least one zero singular value. Near multicollinearity corresponds to a small singular value relative to the largest one.
The associated columns of describe coefficient combinations that the design measures weakly. For full-column-rank ,
Thus, small singular values lead directly to large uncertainty in particular directions.
The spectral condition number is
A large value indicates sensitivity to perturbations. Its interpretation depends on scaling: expressing one predictor in much larger numerical units can produce a large condition number without indicating a substantive redundancy among standardized predictors. Condition indices based on scaled design matrices are therefore often examined alongside other diagnostics.
Statistical uncertainty and numerical instability are related but distinct. Stable algorithms can reduce computational error without adding information to the data. In particular, QR decomposition or singular value decomposition is generally preferable to explicitly forming and inverting , whose spectral condition number is the square of that of .
Common sources
Redundant coding and accounting identities
Exact dependence often results from model construction. Including an intercept together with indicators for every category of a categorical variable creates dependence because the indicators sum to the intercept column. This is commonly called the dummy-variable trap.
Other examples include entering:
- a total together with all of its components;
- the same measurement in two units;
- all proportions in a composition together with an intercept.
These cases usually require an explicit choice of reference category, constraint, or alternative parameterization.
Related measurements and restricted samples
Predictors may measure closely related features, such as several dimensions of body size or multiple indicators of economic activity. They need not be redundant in principle, but a particular sample may contain little independent variation among them.
A restricted sample can also produce strong dependence that would be weaker in a broader population. Repeatedly collecting observations with essentially the same predictor relationships may improve precision slowly without substantially improving the ability to separate effects.
Polynomial terms, interactions, and trends
A predictor and its square can be highly correlated when the predictor occupies a narrow range far from zero. Product terms representing statistical interactions can likewise correlate strongly with their component predictors.
In time series, several predictors may move together because of common trends. Multicollinearity is distinct from problems involving nonstationarity or correlated errors, although these can occur in the same model and require separate treatment.
Diagnosis in practice
No single statistic completely characterizes the problem. A diagnosis can combine:
- Structural inspection: checking identities, duplicate columns, coding conventions, and the relationship between the number of observations and parameters.
- Auxiliary regressions and VIFs: assessing how well each predictor is explained by the others.
- Singular values and condition indices: identifying poorly determined combinations of columns.
- Coefficient uncertainty: examining standard errors, confidence intervals, and joint tests rather than coefficient signs alone.
- Sensitivity analysis: checking how estimates change under reasonable changes to the observations or model specification.
If , full column rank is impossible. Even when , other dependencies can make the design rank deficient. Conversely, a full-rank matrix need not provide precise estimates.
Diagnostics also depend on the model representation. Adding or removing predictors changes VIFs, and changing units can change unscaled condition numbers. Such measures describe the fitted design, not immutable properties of a variable.
Responses and their trade-offs
Collecting informative variation
Experimental design can reduce dependence by varying predictors separately. In observational settings, additional observations are most informative for separating effects when they expand independent predictor variation.
A larger sample drawn under the same full-rank design distribution can improve precision even when the predictor correlations remain unchanged. Additional observations cannot resolve an exact dependence that holds throughout the underlying design.
Reparameterization and centering
Redundant columns can be replaced with an identifiable parameterization. In other cases, it may be more meaningful to estimate a combined effect than to separate effects that the design cannot distinguish.
Centering can reduce correlations between main-effect columns and polynomial or interaction columns, but it does not generally eliminate the underlying information limitation. For an additive model with an intercept, subtracting predictor means does not change predictor correlations or slope VIFs. In polynomial and interaction models, centering changes the interpretation of lower-order coefficients and can improve numerical conditioning without changing the fitted model space.
Regularization and dimension reduction
Ridge regression minimizes an objective of the form
with the intercept usually excluded from the penalty. The penalty stabilizes estimation by shrinking coefficient directions that are weakly supported by the data. This introduces bias and changes the bias–variance trade-off; it does not create new identifying information.
Lasso regression can produce sparse estimates, but its selection among highly correlated predictors may be unstable. Elastic net combines penalties that can be useful for groups of correlated predictors.
Principal component analysis can replace predictors with orthogonal components. Retaining only some components reduces dimension but may discard information relevant to the outcome, because principal components are selected to explain predictor variation rather than outcome variation. The resulting coefficients also refer to combinations of predictors rather than the original variables individually.
Removing predictors
Removing one of several redundant measurements can simplify a predictive model. However, dropping a variable solely because of a high VIF is not a generally valid strategy.
In causal inference, a variable may be required to address confounding. Removing it can change the target interpretation or introduce omitted-variable bias. Whether two variables can be combined or one can be omitted depends on the scientific question and model assumptions, not just their correlation.
Prediction versus interpretation
Multicollinearity is often more damaging to the interpretation of individual coefficients than to prediction. When two predictors nearly move together, their separate coefficients may be unstable while their combined contribution remains well determined.
For a new predictor vector , the conditional variance of the estimated mean response under the classical model is
This can be small when follows the combinations well represented in the observed data, even if some individual coefficient variances are large.
Prediction becomes less secure when new observations depart from those relationships. A model trained where two predictors almost always rise together may provide little information about what happens when one rises while the other stays fixed. Consequently, satisfactory predictive performance within the observed predictor distribution does not establish that separate predictor effects have been precisely estimated.