{
  "id": "multicollinearity",
  "title": "Multicollinearity",
  "lang": "en",
  "source_lang": "en",
  "summary": "Multicollinearity is exact or approximate linear dependence among predictors in a statistical model, limiting the separate estimation of their effects.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Collinearity among predictors"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/multicollinearity/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/multicollinearity/",
      "json": "https://aiwiki.page/zh/multicollinearity.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Multicollinearity is a condition in statistics in which two or more predictors in a statistical model are exactly or approximately related by a linear equation. In linear regression, exact multicollinearity prevents the unique estimation of all regression coefficients, while approximate multicollinearity can make individual coefficient estimates imprecise and sensitive to small changes in the data. It concerns relationships among predictors, not their relationships with the outcome, and does not necessarily prevent accurate prediction."
    },
    {
      "heading": "Mathematical definition",
      "text": "Consider the linear regression model\n\n\\[y=X\\beta+\\varepsilon,\\]\n\nwhere \\(y\\) is an \\(n\\)-dimensional outcome vector, \\(X\\) is an \\(n\\times p\\) design matrix, \\(\\beta\\) contains the regression coefficients, and \\(\\varepsilon\\) contains the errors. The columns of \\(X\\) represent predictors and, when included, an intercept.\n\nExact multicollinearity occurs when there is a nonzero vector \\(a\\) such that\n\n\\[Xa=0.\\]\n\nThe columns of \\(X\\) then fail to satisfy linear independence, and its rank is less than \\(p\\). Equivalently, at least one column is a linear combination of the others.\n\nApproximate multicollinearity, often called near multicollinearity, occurs when some nonzero \\(a\\) produces an \\(Xa\\) that is small relative to the scales of \\(X\\) and \\(a\\). The matrix can retain full column rank while containing little information about certain combinations of coefficients.\n\nThe distinction is important: exact dependence is an algebraic property, whereas the practical severity of approximate dependence depends on predictor scaling, sample size, error variability, and the quantities being estimated.\n\nWhy pairwise correlations are insufficient\n\nA large absolute correlation between two predictors can indicate multicollinearity, but it is not necessary. For example, suppose \\(x_1\\) and \\(x_2\\) are independent, centered predictors with equal variance, and\n\n\\[x_3=x_1+x_2.\\]\n\nThe three columns are exactly dependent even though neither correlation involving \\(x_3\\) equals \\(1\\) in absolute value.\n\nMore generally, a predictor can be closely approximated by a combination of many others without being strongly correlated with any single one. A table of pairwise correlations therefore cannot rule out multicollinearity."
    },
    {
      "heading": "Consequences for coefficient estimation",
      "text": "Exact dependence and non-identification\n\nWhen \\(X\\) has full column rank, the ordinary least squares estimator is\n\n\\[\\hat\\beta=(X^\\mathsf{T}X)^{-1}X^\\mathsf{T}y.\\]\n\nUnder exact multicollinearity, \\(X^\\mathsf{T}X\\) has no inverse, so this expression is undefined.\n\nThe least-squares problem still has solutions, but the coefficient vector is not unique. If \\(Xa=0\\), then\n\n\\[X(\\beta+ta)=X\\beta\\]\n\nfor every real number \\(t\\). Thus, multiple coefficient vectors give identical fitted values. The difficulty is one of statistical identification: the observed design does not distinguish among those coefficient vectors.\n\nAlthough the coefficients are not unique, the least-squares fitted vector is unique because it is the orthogonal projection of \\(y\\) onto the column space of \\(X\\). Some coefficient combinations can also remain identifiable.\n\nFor example, if \\(x_2=2x_1\\), then\n\n\\[\\beta_1x_1+\\beta_2x_2 =(\\beta_1+2\\beta_2)x_1.\\]\n\nThe data can identify \\(\\beta_1+2\\beta_2\\), but not \\(\\beta_1\\) and \\(\\beta_2\\) separately.\n\nSoftware may handle exact dependence by omitting columns, imposing constraints, or selecting a solution through a generalized inverse. Such conventions do not supply information about otherwise unidentified coefficients.\n\nApproximate dependence and uncertainty\n\nSuppose \\(X\\) has full column rank and the errors satisfy\n\n\\[\\operatorname{E}(\\varepsilon\\mid X)=0, \\qquad \\operatorname{Var}(\\varepsilon\\mid X)=\\sigma^2I.\\]\n\nThe conditional covariance matrix of the ordinary least squares estimator is\n\n\\[\\operatorname{Var}(\\hat\\beta\\mid X) =\\sigma^2(X^\\mathsf{T}X)^{-1}.\\]\n\nNear dependence makes some directions in coefficient space weakly determined. The resulting estimates can have large standard errors and wide confidence intervals. Coefficient magnitudes or signs may change substantially when observations are added, removed, or slightly perturbed.\n\nMulticollinearity alone does not create bias in ordinary least squares under the stated zero-conditional-mean assumption. It affects precision rather than automatically shifting estimates systematically away from their true values. It also does not by itself imply endogeneity, which concerns relationships between predictors and the error term.\n\nA model can consequently show strong evidence that a group of predictors matters jointly while individual coefficient tests provide little evidence about their separate contributions."
    },
    {
      "heading": "Variance inflation factors",
      "text": "A common diagnostic is the variance inflation factor (VIF). In an ordinary least squares model with an intercept, regress predictor \\(x_j\\) on all the other predictors, including an intercept. If the auxiliary regression has coefficient of determination \\(R_j^2\\), then\n\n\\[\\operatorname{VIF}_j=\\frac{1}{1-R_j^2}.\\]\n\nUnder the classical constant-error-variance assumptions,\n\n\\[\\operatorname{Var}(\\hat\\beta_j\\mid X) = \\frac{\\sigma^2} {\\sum_{i=1}^{n}(x_{ij}-\\bar x_j)^2} \\operatorname{VIF}_j.\\]\n\nThis separates three influences on precision:\n\n- the error variance \\(\\sigma^2\\);\n- the observed spread of predictor \\(x_j\\);\n- the extent to which the remaining predictors explain \\(x_j\\).\n\nA VIF of \\(1\\) indicates that the centered predictor is orthogonal to the other centered predictors. A VIF of \\(5\\) means that, holding error variance and the predictor’s centered sum of squares fixed, its coefficient variance is five times the corresponding orthogonal-design value. Its standard error is multiplied by \\(\\sqrt{5}\\), not by \\(5\\).\n\nThresholds such as \\(5\\) or \\(10\\) are commonly used as screening conventions, but they are not universal decision rules. A high VIF need not make an estimate unusable if the dataset contains sufficient information, while a low VIF does not guarantee a narrow confidence interval. The reciprocal,\n\n\\[1-R_j^2,\\]\n\nis sometimes called tolerance."
    },
    {
      "heading": "Geometric and numerical interpretation",
      "text": "The singular value decomposition provides a useful description:\n\n\\[X=U\\Sigma V^\\mathsf{T}.\\]\n\nExact multicollinearity corresponds to at least one zero singular value. Near multicollinearity corresponds to a small singular value relative to the largest one.\n\nThe associated columns of \\(V\\) describe coefficient combinations that the design measures weakly. For full-column-rank \\(X\\),\n\n\\[\\operatorname{Var}(\\hat\\beta\\mid X) = \\sigma^2V(\\Sigma^\\mathsf{T}\\Sigma)^{-1}V^\\mathsf{T}.\\]\n\nThus, small singular values lead directly to large uncertainty in particular directions.\n\nThe spectral condition number is\n\n\\[\\kappa_2(X)=\\frac{s_{\\max}}{s_{\\min}}.\\]\n\nA large value indicates sensitivity to perturbations. Its interpretation depends on scaling: expressing one predictor in much larger numerical units can produce a large condition number without indicating a substantive redundancy among standardized predictors. Condition indices based on scaled design matrices are therefore often examined alongside other diagnostics.\n\nStatistical uncertainty and numerical instability are related but distinct. Stable algorithms can reduce computational error without adding information to the data. In particular, QR decomposition or singular value decomposition is generally preferable to explicitly forming and inverting \\(X^\\mathsf{T}X\\), whose spectral condition number is the square of that of \\(X\\)."
    },
    {
      "heading": "Common sources",
      "text": "Redundant coding and accounting identities\n\nExact dependence often results from model construction. Including an intercept together with indicators for every category of a categorical variable creates dependence because the indicators sum to the intercept column. This is commonly called the dummy-variable trap.\n\nOther examples include entering:\n\n- a total together with all of its components;\n- the same measurement in two units;\n- all proportions in a composition together with an intercept.\n\nThese cases usually require an explicit choice of reference category, constraint, or alternative parameterization.\n\nRelated measurements and restricted samples\n\nPredictors may measure closely related features, such as several dimensions of body size or multiple indicators of economic activity. They need not be redundant in principle, but a particular sample may contain little independent variation among them.\n\nA restricted sample can also produce strong dependence that would be weaker in a broader population. Repeatedly collecting observations with essentially the same predictor relationships may improve precision slowly without substantially improving the ability to separate effects.\n\nPolynomial terms, interactions, and trends\n\nA predictor and its square can be highly correlated when the predictor occupies a narrow range far from zero. Product terms representing statistical interactions can likewise correlate strongly with their component predictors.\n\nIn time series, several predictors may move together because of common trends. Multicollinearity is distinct from problems involving nonstationarity or correlated errors, although these can occur in the same model and require separate treatment."
    },
    {
      "heading": "Diagnosis in practice",
      "text": "No single statistic completely characterizes the problem. A diagnosis can combine:\n\n- Structural inspection: checking identities, duplicate columns, coding conventions, and the relationship between the number of observations and parameters.\n- Auxiliary regressions and VIFs: assessing how well each predictor is explained by the others.\n- Singular values and condition indices: identifying poorly determined combinations of columns.\n- Coefficient uncertainty: examining standard errors, confidence intervals, and joint tests rather than coefficient signs alone.\n- Sensitivity analysis: checking how estimates change under reasonable changes to the observations or model specification.\n\nIf \\(p>n\\), full column rank is impossible. Even when \\(p\\leq n\\), other dependencies can make the design rank deficient. Conversely, a full-rank matrix need not provide precise estimates.\n\nDiagnostics also depend on the model representation. Adding or removing predictors changes VIFs, and changing units can change unscaled condition numbers. Such measures describe the fitted design, not immutable properties of a variable."
    },
    {
      "heading": "Responses and their trade-offs",
      "text": "Collecting informative variation\n\nExperimental design can reduce dependence by varying predictors separately. In observational settings, additional observations are most informative for separating effects when they expand independent predictor variation.\n\nA larger sample drawn under the same full-rank design distribution can improve precision even when the predictor correlations remain unchanged. Additional observations cannot resolve an exact dependence that holds throughout the underlying design.\n\nReparameterization and centering\n\nRedundant columns can be replaced with an identifiable parameterization. In other cases, it may be more meaningful to estimate a combined effect than to separate effects that the design cannot distinguish.\n\nCentering can reduce correlations between main-effect columns and polynomial or interaction columns, but it does not generally eliminate the underlying information limitation. For an additive model with an intercept, subtracting predictor means does not change predictor correlations or slope VIFs. In polynomial and interaction models, centering changes the interpretation of lower-order coefficients and can improve numerical conditioning without changing the fitted model space.\n\nRegularization and dimension reduction\n\nRidge regression minimizes an objective of the form\n\n\\[\\|y-X\\beta\\|_2^2+\\lambda\\|\\beta\\|_2^2, \\qquad \\lambda>0,\\]\n\nwith the intercept usually excluded from the penalty. The penalty stabilizes estimation by shrinking coefficient directions that are weakly supported by the data. This introduces bias and changes the bias–variance trade-off; it does not create new identifying information.\n\nLasso regression can produce sparse estimates, but its selection among highly correlated predictors may be unstable. Elastic net combines penalties that can be useful for groups of correlated predictors.\n\nPrincipal component analysis can replace predictors with orthogonal components. Retaining only some components reduces dimension but may discard information relevant to the outcome, because principal components are selected to explain predictor variation rather than outcome variation. The resulting coefficients also refer to combinations of predictors rather than the original variables individually.\n\nRemoving predictors\n\nRemoving one of several redundant measurements can simplify a predictive model. However, dropping a variable solely because of a high VIF is not a generally valid strategy.\n\nIn causal inference, a variable may be required to address confounding. Removing it can change the target interpretation or introduce omitted-variable bias. Whether two variables can be combined or one can be omitted depends on the scientific question and model assumptions, not just their correlation."
    },
    {
      "heading": "Prediction versus interpretation",
      "text": "Multicollinearity is often more damaging to the interpretation of individual coefficients than to prediction. When two predictors nearly move together, their separate coefficients may be unstable while their combined contribution remains well determined.\n\nFor a new predictor vector \\(x_0\\), the conditional variance of the estimated mean response under the classical model is\n\n\\[\\operatorname{Var}(x_0^\\mathsf{T}\\hat\\beta\\mid X) = \\sigma^2x_0^\\mathsf{T}(X^\\mathsf{T}X)^{-1}x_0.\\]\n\nThis can be small when \\(x_0\\) follows the combinations well represented in the observed data, even if some individual coefficient variances are large.\n\nPrediction becomes less secure when new observations depart from those relationships. A model trained where two predictors almost always rise together may provide little information about what happens when one rises while the other stays fixed. Consequently, satisfactory predictive performance within the observed predictor distribution does not establish that separate predictor effects have been precisely estimated."
    }
  ],
  "links": [
    "statistics",
    "linear-regression",
    "matrix",
    "linear-independence",
    "matrix-rank",
    "linear-combination",
    "correlation",
    "ordinary-least-squares",
    "inverse-matrix",
    "statistical-identification",
    "orthogonal-projection",
    "covariance-matrix",
    "standard-error",
    "confidence-interval",
    "estimator-bias",
    "endogeneity",
    "coefficient-of-determination",
    "singular-value-decomposition",
    "condition-number",
    "numerical-stability",
    "qr-decomposition",
    "statistical-interaction",
    "time-series",
    "experimental-design",
    "ridge-regression",
    "bias-variance-tradeoff",
    "lasso-regression",
    "elastic-net",
    "principal-component-analysis",
    "causal-inference",
    "confounding"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}