Double descent is a pattern in machine learning in which a model’s prediction error on unseen data first decreases, then increases, and then decreases again as model capacity grows. The peak commonly occurs near the interpolation threshold, where the learning procedure first becomes able to fit the training data exactly or nearly exactly. The phenomenon extends the familiar U-shaped account of overfitting: models beyond the interpolation threshold can sometimes generalize better than smaller models, despite having enough flexibility to fit noise. Related patterns occur as training duration or dataset size changes, although double descent is not universal.
The risk curve and interpolation threshold
In supervised learning, a model is fitted to observed input–output pairs and evaluated on new examples from a specified probability distribution. Its population risk is
where is the fitted predictor and is a loss function. Performance measured on an independent test set estimates this risk. Double-descent curves may show risk conditional on a particular training sample or risk averaged over training samples and other sources of randomness; these need not have identical shapes.
The conventional U-shaped curve describes an initial reduction in error as a model becomes expressive enough to capture useful structure, followed by increasing error as it becomes more sensitive to the particular sample. Double descent adds a further regime:
- Underparameterized regime: Limited capacity can produce underfitting. Increasing capacity often improves prediction, although error can begin to rise before interpolation.
- Interpolation threshold: The fitted model first achieves zero or nearly zero training error. Predictions may become especially sensitive to noise and small changes in the sample.
- Overparameterized regime: Many models can interpolate the observations. The training procedure’s choice among them can yield lower test error as capacity increases further.
Interpolation here means fitting the observed targets, not necessarily estimating the underlying relationship accurately. For classification, zero training classification error is also distinct from zero training loss: cross-entropy, for example, can continue decreasing after every training example is classified correctly.
The threshold is not generally defined by the number of parameters equaling the number of examples. That correspondence holds in particular linear models under rank assumptions. In neural networks, architecture, data structure, the optimization method, and the definition of training error all influence where interpolation becomes possible.
Relation to the bias–variance account
The classical bias–variance tradeoff explains prediction error in terms of systematic approximation error, sensitivity to the training sample, and irreducible observation noise. A common interpretation is that greater model flexibility reduces bias but increases variance, producing an optimal intermediate complexity.
Double descent does not invalidate the bias–variance decomposition. Instead, it shows that variance need not increase monotonically with parameter count, and that parameter count alone is an incomplete description of a fitted model’s effective complexity.
Beyond interpolation, increasing the number of parameters changes the set of available solutions. A learning procedure may select a solution with a smaller norm, greater smoothness, or another useful structural property. These properties depend on the procedure’s inductive bias, not simply on its ability to fit the training observations.
A linear-regression example
Linear regression provides an analytically tractable setting. Let be a design matrix, with training examples and features, and suppose
where is the true coefficient vector and is observation noise.
When and has full column rank, ordinary least squares has a unique solution. When and has full row rank, infinitely many coefficient vectors fit any target vector exactly. A standard choice is the minimum-Euclidean-norm interpolator,
Near , small singular values of can cause the fitted coefficients to amplify observation noise. Farther into the overparameterized regime, this amplification can decrease.
A precise illustration uses independent Gaussian feature vectors with covariance , independent noise of variance , and a sequence of true coefficient vectors whose squared norms approach . As grow with , the excess prediction risk of least squares or its minimum-norm extension approaches
Here “excess” excludes the irreducible noise in a fresh target. The expression is singular at when . On the overparameterized side, the noise contribution decreases with , while the signal component lost in unobserved directions increases.
This example demonstrates an interpolation peak and a second descent, but it does not by itself produce an initial descent: below interpolation, all relevant features are already present and the displayed risk increases. A complete double-descent curve can arise when increasing model size also reduces approximation error, as in suitably chosen feature-expansion models. Thus an interpolation peak and the entire double-descent shape are related but distinct observations.
Why overparameterized models can improve
Several mechanisms help explain the second descent.
Selection among interpolating solutions. Minimizing training error does not uniquely determine a predictor once many exact fits exist. Explicit regularization, such as a penalty on coefficient magnitude, can distinguish solutions. Even without an explicit penalty, an optimization procedure can favor particular solutions through implicit regularization. For example, gradient descent initialized at zero selects the minimum-norm solution in overparameterized linear least squares under suitable convergence conditions.
Reduced noise amplification. In linear models and some kernel methods, predictions depend on the spectrum of a data-dependent matrix. Poorly conditioned directions near interpolation can strongly amplify noise. Increasing model size can alter that spectrum and reduce the amplification.
Distribution of fitted noise across directions. In some high-dimensional settings, an interpolator can fit noisy targets largely through directions that have little influence on prediction for typical new inputs. Under suitable conditions, this permits benign overfitting: exact fitting of noisy training data alongside low excess prediction error. Benign overfitting is not synonymous with double descent. The former concerns the quality of interpolating estimators; the latter describes a curve as some experimental variable changes.
Learned representations. In deep learning, increasing model size can also improve representation learning. Consequently, explanations derived from fixed-feature linear regression illuminate some behavior but do not fully account for trained neural networks.
Main variants
Model-wise double descent
Model-wise double descent varies capacity while holding the dataset and training protocol as comparable as possible. Capacity may be changed through neural-network width, the number of fitted features, or another model-family parameter.
Such comparisons require care. Changing width can also change optimization, suitable learning rates, and the strength of effective regularization. A curve therefore reflects the complete training procedure, not an architecture-independent property of parameter count.
Epoch-wise double descent
Epoch-wise double descent varies training duration. Test error may fall during early learning, rise as the model fits more sample-specific details, and fall again with longer training.
The horizontal axis is training time rather than model size. The pattern can complicate interpretations of early stopping, but longer training does not necessarily produce a second improvement. Its occurrence depends on the model, data, optimizer, and regularization.
Sample-wise double descent
Sample-wise double descent varies the amount of training data while keeping the model and other experimental choices fixed. In certain regimes, adding examples moves a system toward an interpolation threshold and temporarily worsens test performance; adding still more examples improves it again.
This is not a general claim that additional data are harmful. It is a nonmonotonic behavior of particular estimators under particular conditions. Changes in sample size can also change optimization difficulty and the effective strength of regularization.
Historical development
Nonmonotonic prediction error near interpolation was studied in statistical learning and neural-network research before the term double descent became established.
In 2019, Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal popularized the term in an account connecting the classical bias–variance picture with the behavior of highly overparameterized models. Subsequent work developed precise risk calculations for high-dimensional linear regression, random-feature models, and kernel methods.
Research by Preetum Nakkiran and collaborators, circulated in 2019 and presented in 2020, documented double-descent behavior in modern deep-learning systems and examined model-wise, epoch-wise, and sample-wise forms. These investigations helped frame interpolation as an important transition in the study of generalization, rather than treating zero training error as sufficient evidence of poor prediction.
Scope, limitations, and interpretation
Double descent is an observed and mathematically established phenomenon in specified settings, not a universal law of machine learning. Its appearance and magnitude depend on label noise, feature covariance, signal structure, loss, optimizer, regularization, and the experimental variable being changed. Strong regularization can smooth or remove the interpolation peak. Some systems show monotonic improvement, broad plateaus, or multiple peaks instead of a simple double descent.
The second descent need not outperform the best underparameterized model. Nor does being far beyond interpolation guarantee low risk: an estimator can remain inaccurate because of poor features, unfavorable data geometry, or substantial noise.
Evaluation choices also matter. A peak in squared prediction error may be less apparent in classification error, and a finite test set may give a noisy estimate of either. Repeated use of test results for model selection can compromise their independence; validation sets serve a different role from final test evaluation.
Most elementary analyses assume that training and test observations come from the same distribution. Low test error under that assumption does not establish robustness to distribution shift or adversarial examples.
Double descent is also distinct from a neural scaling law. Scaling laws describe systematic relationships between performance and resources such as model size, data, or computation, often over ranges where error decreases smoothly. Double descent specifically concerns a nonmonotonic transition, commonly associated with interpolation.