Stacked generalization is an ensemble-learning method in which the predictions of several models become inputs to another model that learns how to combine them. The component models are called base learners or level-0 models, and the combining model is called a meta-learner or level-1 model. Used primarily in supervised learning, stacking aims to improve predictive generalization by exploiting differences in the strengths and errors of its components. Its defining feature is a learned combination rule, rather than a fixed vote or average.
Principle and architecture
Suppose a dataset contains input–target pairs , and base learners produce predictions . A stacked predictor has the form
where is the meta-learner.
The base learners may use different algorithms, feature representations, subsets of features, or hyperparameter settings. For example, a stack might combine a random forest, a gradient-boosted model, and an artificial neural network. Architectural diversity is useful only insofar as it produces complementary predictive information: different algorithms can still make very similar errors.
A simple meta-learner forms a linear combination,
The weights may be unconstrained, restricted to be nonnegative, or required to sum to one. More flexible meta-learners can learn nonlinear interactions or input-dependent combinations. Some implementations also pass the original features to the meta-learner:
This can help the combiner identify circumstances in which particular models perform well, but it also increases its capacity and potential for overfitting.
Stacking does not require the base learners to be weak models, and the models need not be trained jointly.
Training with out-of-fold predictions
The central training problem is how to create realistic inputs for the meta-learner. Predictions made on the same examples used to fit a base model can be substantially more accurate than its predictions on unseen examples. Training a combiner on those in-sample predictions can therefore teach it relationships that do not hold at deployment.
The standard solution is to generate out-of-fold predictions using cross-validation.
Constructing the meta-training data
Partition the development data into disjoint folds . For each base-learning procedure and each fold :
- Train the base learner on all folds except .
- Predict the examples in .
- Store those predictions in their original row positions.
For , define
where denotes the model trained without fold . The resulting meta-feature matrix is
Each row contains predictions for an example that was excluded from the corresponding base-model fits. The meta-learner is then trained to predict from .
For vector-valued base outputs, such as multiclass probabilities, each learner contributes multiple columns rather than a single column.
Fitting models for deployment
After constructing the out-of-fold features, a common procedure refits each base learner on all available development data. At inference time, these full-data models produce the meta-features supplied to the already fitted combiner.
Another approach retains the fold-specific models and averages their predictions before passing them to the combiner. These approaches differ in computational cost and in the distribution of the meta-features they produce.
In either case, there can be a mismatch between meta-training and deployment: out-of-fold predictions come from models trained on only part of the data, while deployment predictions may come from full-data models or averages of several models. This mismatch is one reason why out-of-fold construction alone does not guarantee an improvement.
Prediction types and objectives
Continuous targets
For continuous-target prediction, each base model commonly contributes one numerical prediction. A linear combiner with regularization is often useful when these predictions are strongly correlated.
For squared-error prediction, the value of combining models depends partly on their error covariance. Let , and suppose a weighted combination has weights summing to one. Its error is
Consequently,
This illustrates why individual accuracy is not the only consideration: a somewhat less accurate model may contribute useful information if its errors differ from those of stronger models. Lower error covariance does not by itself guarantee lower overall prediction error, because bias and weight estimation also matter.
Classification
For binary classification, each base model may provide a class probability, decision score, or class label. Probabilities and scores usually retain more information than hard labels. A logistic regression model can serve as a relatively simple meta-learner.
For multiclass classification, a model may supply one probability per class. Class ordering must be consistent across learners. Because class probabilities sum to one, retaining every probability column introduces linear dependence; whether this matters depends on the combiner.
The training objective should match the intended predictive task. A stack optimized for probability-based loss need not be the best stack for ranking, thresholded accuracy, or another evaluation criterion. Base-model probabilities also need not be equally well calibrated, and stacking does not automatically ensure calibrated final probabilities.
Evaluation and data leakage
Out-of-fold predictions are a training mechanism, not an independent evaluation of the complete stack. Once the meta-learner has been fitted to those predictions and their targets, its performance on that same meta-training data is an in-sample result.
An honest estimate of final performance requires either:
- an untouched test set excluded from every fitting and selection decision; or
- an outer evaluation loop in which the entire stacking procedure is rebuilt using only each outer training partition.
When cross-validation is used both to construct or select the stack and to evaluate it, nested cross-validation provides the appropriate separation. The inner procedure creates meta-features and performs model selection; the outer procedure evaluates predictions on data unseen by the entire fitted stack.
Data leakage can occur even when the base-model fitting itself excludes the held-out fold. Learned preprocessing, including feature scaling, imputation, and feature selection, must also respect fold boundaries. Selecting hyperparameters using a held-out example's target and then treating its prediction as fully out of sample can likewise compromise the separation.
The splitting scheme must reflect the data structure:
- Grouped observations: related examples, such as repeated records for the same entity, may need to remain in the same fold.
- Time-ordered observations: predictions intended to represent future deployment should be generated without training on future observations.
- Rare classes: stratified folds can help maintain usable class representation.
With forward-looking temporal splits, not every observation necessarily receives a valid out-of-fold prediction. The meta-training dataset may therefore exclude an initial period.
Relationship to other ensemble methods
Averaging and voting. These use predetermined aggregation rules. Stacking learns the rule from labeled data, although a learned stack can ultimately resemble a weighted average.
Bagging. Bagging typically fits similar learners to resampled datasets and aggregates them by averaging or voting. Stacking instead focuses on learning how to combine model outputs. Bagged models can themselves be components of a stack.
Boosting. Boosting builds an ensemble sequentially, with later components responding to earlier errors or an evolving optimization objective. Stacking generally trains base learners separately and then fits a combiner to their predictions.
Blending. In common usage, blending trains the combiner on predictions for a dedicated holdout subset rather than on cross-validated out-of-fold predictions. Terminology varies. A holdout design is simpler, but reserves data that might otherwise help fit the base models.
Knowledge distillation. Distillation trains a student model to reproduce outputs from a teacher or ensemble. Stacking normally retains the base models and uses their predictions at inference time; distillation can instead replace the ensemble with a single deployable model.
History
David H. Wolpert introduced stacked generalization in a 1992 paper of that title. The formulation used predictions generated under a cross-validation scheme to learn how to combine generalizers.
Leo Breiman's 1996 work on stacked regressions developed an influential formulation for continuous targets, including combinations with nonnegative coefficients. Subsequent work extended cross-validated combination to broader libraries of candidate learners.
One related framework is the Super Learner, introduced by Mark van der Laan, Eric Polley, and Alan Hubbard in 2007. It uses cross-validation to select a combination from a specified candidate library. Its theoretical performance guarantees depend on assumptions about the loss, candidate procedures, and asymptotic setting; they do not mean that every finite-sample stack outperforms its strongest component.
Applications and limitations
Stacking is used in predictive modeling where multiple models capture complementary structure, including tabular prediction, recommendation, and competition ensembles. It can combine models built from different representations of the same examples, such as structured features and text-derived predictions.
Its principal costs and limitations include:
- Training expense. With base learners and folds, generating meta-features requires approximately base-model fits, before full-data refitting or hyperparameter searches.
- Inference expense. Deployment generally requires running every retained base model before the combiner can produce a prediction.
- Unstable combinations. Highly correlated predictions can make unconstrained weights unstable, particularly with limited meta-training data.
- Limited complementary information. Combining nearly identical predictions may add complexity without meaningful predictive benefit.
- Operational coupling. Changes to base models, feature pipelines, or output conventions can invalidate the meta-features expected by the combiner.
- Distribution shift. Relationships among model errors learned during training may not persist in a changed deployment population.
A stack can perform worse than its best component. Its value is therefore established by evaluating the complete training-and-prediction procedure against strong individual models and simpler aggregation baselines under the same data-access conditions.