Degrees of freedom are the number of independent quantities needed to describe variation within a system after its constraints have been taken into account. In mathematics, this often means the dimension of a space of admissible solutions; in statistics, it commonly describes the independent variation remaining after parameters have been estimated. Mechanical usage concerns independent coordinates specifying a configuration. These meanings share a dimensional interpretation, although statistical “effective degrees of freedom” generalize the idea beyond an integer count. (ocw.mit.edu)
Mathematical interpretation
In linear algebra, degrees of freedom can be expressed precisely through the dimension of a vector space. Consider a consistent system of linear equations
[ Ax=b, ]
where (x) has (n) components and the matrix (A) has rank (r). Its solutions have the form (x=x_0+v), where (x_0) is one particular solution and (Av=0). The rank–nullity theorem gives (n-r) independent directions of variation. Thus the solution set has (n-r) degrees of freedom, regardless of how many equations are written: redundant equations impose no additional independent restrictions. (ocw.mit.edu)
For example, three real numbers constrained by (x_1+x_2+x_3=10) have two degrees of freedom. Once two are chosen, the third is determined. Adding twice the same equation does not reduce this count. Here “independent” means independently specifiable coordinates, not necessarily statistical independence between observed quantities. This distinction matters when interpreting statistical residuals, which may satisfy linear constraints without being independent random variables. (ocw.mit.edu)
Sample variance and the loss of one degree
A central statistical example is estimating variance from (n) observations. Their deviations from the sample mean satisfy
[ \sum_{i=1}^{n}(x_i-\bar{x})=0. ]
Consequently, only (n-1) deviations are independently specifiable. Estimating the mean imposes one constraint on the residual vector, leaving (n-1) degrees of freedom. (ocw.mit.edu)
For independent, identically distributed observations with finite variance (\sigma^2), the sample variance
[ s^2=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2 ]
is unbiased: its expected value equals (\sigma^2). Using (n-1) rather than (n) is Bessel’s correction. Normality is not required for this unbiasedness result. The correction concerns variance; taking its square root does not generally produce an unbiased estimator of the population standard deviation. (arxiv.org)
Degrees of freedom in inference
Degrees of freedom also parameterize a probability distribution used in inference. Under independent sampling from a normal distribution,
[ T=\frac{\bar{x}-\mu}{s/\sqrt n} ]
follows Student’s t-distribution with (n-1) degrees of freedom. This distribution accounts for uncertainty introduced by estimating the population standard deviation. Its quantiles determine the width of a conventional confidence interval for the mean. (iso.org)
The chi-square distribution likewise appears when squared Gaussian variation is measured after projection onto a subspace. Its degrees of freedom correspond to that subspace’s dimension. Degrees of freedom therefore connect geometric restrictions with the reference distributions used in statistical hypothesis testing, rather than merely labeling sample size. (arxiv.org)
For a two-way contingency table with (r) rows and (c) columns, Pearson’s test of independence conventionally uses ((r-1)(c-1)) degrees of freedom. With row and column totals fixed, specifying the upper-left ((r-1)\times(c-1)) block determines the remaining cells. The chi-square reference distribution is approximate, and its adequacy depends on the sampling conditions and expected cell counts. (itl.nist.gov)
Regression and residual variation
In linear regression fitted by ordinary least squares, a fixed design matrix (X) of rank (r) provides (r) model degrees of freedom. The fitted response is an orthogonal projection onto the column space of (X); residuals occupy its orthogonal complement, with (n-r) degrees of freedom. For a full-rank model with (p) coefficients, this becomes (n-p). An estimated intercept counts as a coefficient. (arxiv.org)
A straight-line model with an intercept and slope therefore has (n-2) residual degrees of freedom. Dividing its residual sum of squares by (n-2) gives the usual residual mean square. More generally, division by (n-r) yields an unbiased error-variance estimate when the linear mean model is correct and errors have a common variance and are uncorrelated. Exact chi-square inference additionally requires Gaussian errors. (online.stat.psu.edu)
Effective degrees of freedom
In machine learning and statistical smoothing, parameter count may poorly describe how strongly a procedure adapts to data. For a linear fitting rule (\hat y=Sy), with (S) fixed independently of the observed response, a common effective degrees-of-freedom definition is
[ \operatorname{df}_{\mathrm{eff}}=\operatorname{tr}(S), ]
where (\operatorname{tr}) is the matrix trace. Ordinary least squares recovers the model rank. With regularization, as in ridge regression, shrinking directions of variation can produce fractional effective degrees of freedom. This quantity is not automatically an exact residual dimension or a reference-distribution parameter. (arxiv.org)
For more general fitting procedures with independent, equal-variance errors, a related definition is
[ \operatorname{df}_{\mathrm{eff}} =\frac{1}{\sigma^2}\sum_i\operatorname{Cov}(y_i,\hat y_i). ]
It measures adaptation through covariance and is connected to optimism in training error. Adaptive model selection can make it differ substantially from the number of retained coefficients; it need not behave like an ordinary dimension. (arxiv.org)
Mechanical usage
In classical mechanics and robotics, degrees of freedom count independent configuration coordinates. A free rigid body has three in a plane—two translations and one rotation—and six in three-dimensional space—three translations and three rotations. Joints and other independent constraints reduce the allowable configurations. These are configuration degrees of freedom, not counts of statistical observations or fitted coefficients. (modernrobotics.northwestern.edu)