The bootstrap method is a family of computational techniques in statistics for estimating how a result would vary across repeated samples. Its central operation is to construct artificial datasets from the observed data, recalculate a statistic on each dataset, and use the resulting variation to approximate the statistic’s sampling distribution. The best-known version draws observations independently, with replacement, from the original sample. Other versions simulate from a fitted probability model or resample residuals, clusters, or blocks of consecutive observations.
The bootstrap is important because uncertainty can be difficult to calculate analytically, even when a statistic itself is easy to compute. It provides estimates of standard error, bias, and confidence intervals for quantities ranging from a simple average to the output of a complicated modeling procedure. Its flexibility does not make it assumption-free: the resampling scheme must reproduce the important features of the original sampling process, and some statistics have distributions that ordinary bootstrapping cannot approximate correctly.
Statistical formulation and the resampling mechanism
Suppose are independent observations from an unknown probability distribution , and the target is a population quantity . An estimate is computed from the observed sample. The inferential problem is to determine how would change if another sample of size were drawn from . Because is unknown, repeated sampling from the actual population is usually unavailable.
The ordinary nonparametric bootstrap substitutes the empirical distribution for . This distribution assigns probability to each observed data record, counting repeated values according to their multiplicity. Its cumulative distribution function is
where is an indicator function. A bootstrap sample consists of independent draws from , conditional on the original data. The statistic is then recomputed as . This sampling with replacement is repeated times, producing .
Conditional on the observed data, the bootstrap standard-error estimate is
The corresponding bias estimate is . For a vector-valued estimate, the same calculation yields a bootstrap covariance matrix, describing both marginal uncertainty and dependence among components.
The sample mean provides an exact illustration. Conditional on the data,
Here and refer to bootstrap randomness while holding the data fixed. If is the usual sample variance with denominator , the bootstrap variance is , rather than exactly . Thus even a familiar statistic exhibits a small finite-sample difference between the bootstrap answer and an unbiased analytic variance estimate.
Replacement is essential. Sampling all observations without replacement merely reorders them and leaves any order-invariant statistic unchanged. With replacement, some observations occur repeatedly and others are absent. The expected number of distinct sampled records is
which approaches approximately . These repetitions do not represent additional independent information. They create a computational approximation to sampling variability using the information contained in the original sample.
Why the bootstrap works
The bootstrap rests on a substitution principle: when the empirical distribution approximates the population distribution adequately, the behavior of a statistic under empirical resampling may approximate its behavior under population sampling. The law of large numbers supports the empirical distribution’s accuracy, but that fact alone is insufficient. The statistic must also respond suitably to perturbations of the underlying distribution.
A typical validity statement compares the distribution of a centered, scaled estimator with its conditional bootstrap counterpart. For a root- estimator, one seeks
This expresses bootstrap consistency in a common setting with a continuous limiting distribution. The left-hand distribution is conditional on the observed data, whereas the right-hand distribution describes repeated sampling from the population. The convergence is convergence in probability over possible original datasets.
For a sample mean with finite, nonzero population variance, the central limit theorem gives an asymptotically normal sampling distribution. The bootstrap reproduces this leading behavior because the empirical variance converges to the population variance and no single observation dominates asymptotically. Similar arguments extend to many smooth functions of averages, regular regression coefficients, and other estimators admitting a representation
where , has finite variance, and the remainder is negligible relative to . The function , often called an influence function in this context, describes the estimator’s leading response to individual observations.
Smoothness need not mean that the statistic is an elementary differentiable formula. A sample median, for example, is nonsmooth as a function of ordered data, yet its bootstrap distribution can be valid when the population has a positive, continuous density near the population median. What matters is the regularity of the statistical problem, not merely the appearance of the calculation.
Bootstrap approximations can sometimes improve on a first-order normal approximation. Under stronger smoothness and moment conditions, suitably studentized bootstrap procedures capture skewness and other terms appearing in an asymptotic expansion. This can reduce coverage or rejection-probability error beyond the order achieved by a basic normal approximation. Such improvements are not universal: they depend on the statistic, the resampling scheme, and the inferential construction.
Finally, theoretical validity concerns increasing original sample size, not merely increasing the number of bootstrap replicates. Taking to infinity makes the simulated bootstrap distribution arbitrarily accurate for the observed dataset. It does not establish that this conditional distribution approximates the actual sampling distribution well. Approximation error from the statistical model and simulation error from finite are distinct.
Confidence intervals, bias correction, and studentization
Several confidence-interval constructions use the same bootstrap replicates but interpret them differently. Let denote the estimated -quantile of the bootstrap estimates. A percentile interval with nominal coverage is
Its simplicity is attractive, and it is transformation-respecting: for a strictly increasing transformation , transforming the interval endpoints gives the percentile interval obtained by bootstrapping , subject to the quantile convention used. It also respects some natural parameter restrictions because its endpoints come from actual bootstrap estimates. Nevertheless, bias and asymmetry can cause substantial coverage error.
The basic, or reverse-percentile, interval instead uses the bootstrap distribution of the estimation error:
The reversal follows from approximating the distribution of by that of . Unlike the percentile interval, this interval is not generally invariant under nonlinear transformations and can extend outside the parameter’s possible range.
A normal-based bootstrap interval takes the form
where is a standard normal quantile. It uses bootstrapping only to estimate the standard error. A bias-adjusted version replaces the center by . Such correction may reduce bias but need not improve mean squared error, because estimating the correction introduces additional variability.
The studentized bootstrap, also called bootstrap-, constructs replicate statistics
If denotes a quantile of these statistics, the interval is
Studentization adjusts for the possibility that uncertainty varies with the estimated parameter. The replicate-specific standard errors can come from analytic formulas or an inner bootstrap. Nested resampling may be computationally expensive, and unstable or zero replicate standard errors can make the method unreliable.
The bias-corrected and accelerated, or BCa, interval modifies percentile levels using a median-bias adjustment and an acceleration parameter . The adjusted probability corresponding to a nominal probability is
where is the standard normal distribution function. The quantity is estimated from the proportion of bootstrap estimates below . Acceleration is commonly estimated by the jackknife, which recalculates the statistic after deleting each observation in turn. BCa intervals can provide higher-order accuracy in regular problems and preserve transformation behavior, but discrete distributions, unstable deletion estimates, and extreme adjusted probabilities can impair their performance.
A confidence interval concerns repeated-sampling coverage of a fixed population parameter. The fraction of bootstrap estimates inside an interval is not, by itself, a posterior probability that the parameter lies there.
Parametric, regression, and weighted variants
The nonparametric bootstrap treats the empirical distribution as a fitted population. A parametric bootstrap instead fits a model , generates new datasets from that model, and repeats the entire estimation procedure. For example, observations assumed to follow a Poisson distribution can be resampled from a Poisson distribution whose rate is the observed mean. Parametric bootstrapping can simulate values absent from the original data, but its adequacy depends on the fitted model describing the relevant distributional features.
Parametric simulation is especially useful when a model has a complicated finite-sample sampling distribution. A statistic involving variance components, latent variables, or a nonlinear fitted curve may be straightforward to calculate repeatedly even when an analytic distribution is unavailable. Parameters estimated by maximum likelihood ordinarily must be re-estimated in every replicate. Simulating datasets while leaving a nuisance-parameter estimate fixed during the subsequent analysis generally represents a different inferential procedure.
For linear regression, the distinction between pairs resampling and residual resampling is fundamental. Pairs resampling draws complete rows , preserving the observed association between predictors and outcomes. It naturally corresponds to a random-design interpretation in which the rows are independent observations from a joint population. Resampling predictor columns and outcomes independently would destroy their relationship.
Residual resampling starts from a fitted model
It holds the design points fixed, samples suitably centered fitted residuals, and constructs . Pooling residuals treats their distributions as interchangeable. This is appropriate only under corresponding assumptions about error distributions; in particular, ordinary pooled residual resampling does not preserve heteroskedasticity.
The wild bootstrap addresses heteroskedastic regression errors through pseudo-residuals such as
where is a centered or leverage-adjusted residual and the independent multipliers have mean zero and variance one. Because the residual magnitude remains attached to its design point, the procedure retains an estimate of observation-specific error scale. Choices of multiplier distribution and residual adjustment affect finite-sample behavior.
Weighted bootstraps offer another representation. Ordinary resampling can be expressed by assigning multinomial counts to the original observations, with total count . Rather than duplicating rows, software can compute the statistic using these counts as weights. Other weighted procedures use different random weights and require their own justification.
The Bayesian bootstrap, introduced by Donald Rubin in 1981, assigns random probability weights with a Dirichlet distribution to the observed values. It produces a Bayesian distribution over functionals of these weighted empirical distributions. Although computationally related to ordinary bootstrapping, its interpretation belongs to Bayesian inference rather than directly to repeated-sampling approximation. Neither method should be confused with simulating future observations from a posterior predictive distribution.
Dependence, clustering, and complex sampling designs
Ordinary observation-level resampling assumes that individual records are the appropriate independent sampling units. That assumption fails for many datasets. Repeated measurements from one person, pupils in the same school, and consecutive measurements in a time series can be dependent. Resampling such records independently usually destroys dependence and misrepresents uncertainty, often producing standard errors that are too small.
For independent clusters, a cluster bootstrap samples entire clusters with replacement and retains the observations inside each selected cluster. The relevant asymptotic information may then depend primarily on the number of independent clusters, rather than the total number of individual records. A dataset with thousands of measurements but only a few clusters cannot generally obtain reliable large-sample cluster inference merely by increasing the number of bootstrap replicates.
The resampling procedure must also distinguish the estimand from the sampling unit. For example, an average over individuals and an average of cluster means need not target the same population quantity when cluster sizes vary. Repeated selection of a cluster must preserve its identity as a separately sampled copy when the analysis includes cluster-specific effects. Stratified sampling similarly calls for resampling within strata rather than mixing all observations indiscriminately.
For serially dependent observations, the block bootstrap resamples groups of adjacent values. The moving-block bootstrap forms overlapping blocks of length , samples blocks, and concatenates them to create a series of approximately the original length. A circular version permits blocks to wrap around the end of the observed series. The stationary bootstrap uses blocks with random, geometrically distributed lengths and circular starting positions.
Block length controls a genuine approximation tradeoff. Short blocks retain little long-range dependence but allow many combinations; long blocks preserve more dependence but provide fewer effectively distinct resampling units. Standard consistency results commonly require to increase with while tends to zero, together with assumptions that dependence weakens sufficiently over time. There is no universally correct finite-sample block length independent of the statistic and dependence structure.
A model-based alternative fits a time-series model, resamples innovations, and regenerates complete series. For an autoregressive model, this preserves dependence through the fitted recurrence rather than by copying observed blocks. Its performance depends on model specification, treatment of initial conditions, and whether parameter estimation is repeated.
Spatial data and multistage surveys require analogous attention. Spatial blocks must reflect the relevant dependence geometry, while survey bootstraps may need to reproduce strata, primary sampling units, unequal selection probabilities, and finite-population corrections. A census of a fixed population, a simple random sample, and a multistage sample do not have the same design-based uncertainty. “Bootstrap” identifies a computational strategy, not a single resampling design valid for all of them.
Hypothesis testing, prediction, and model stability
Bootstrap hypothesis tests approximate the distribution of a test statistic under a null hypothesis. This requires more than repeatedly sampling the observed data unchanged. If the observed data reflect an alternative hypothesis, ordinary empirical resampling will usually reproduce that departure rather than impose the null.
For a test that a population mean equals , one possible construction recenters observations as , then resamples the recentered values. For a parametric likelihood-ratio test, a corresponding construction fits the model under the null, generates datasets from that restricted fit, and refits both restricted and unrestricted models to each generated dataset. The bootstrap then estimates how often the simulated statistic is at least as extreme as the observed statistic. The validity of the procedure depends on the null model and the behavior of the statistic; simulation does not automatically resolve nonregular testing problems.
A permutation test uses a different principle. It rearranges labels or assignments according to a null symmetry or a known randomization design. Under appropriate exchangeability or randomization conditions, such a test can have exact finite-sample properties. Ordinary bootstrap tests generally rely on approximation instead. The fact that both methods create many rearranged datasets does not make their justifications interchangeable.
Prediction introduces another distinction. A confidence interval for a fitted mean describes uncertainty about the mean response. A prediction interval for a new outcome must also account for the new observation’s unexplained variation. Bootstrapping fitted parameters alone therefore does not generally produce a predictive interval. Model-based predictive simulations can combine parameter refitting with draws of future errors or outcomes, using a construction appropriate to the prediction target.
In machine learning, bootstrapping is used both to examine model instability and to create ensembles. Bagging, short for bootstrap aggregating, fits a prediction rule to many bootstrap samples and averages numerical predictions or combines classifications. Averaging can reduce the variance of unstable procedures such as decision trees. This is a change to the prediction algorithm itself, not merely an uncertainty estimate attached to a fixed algorithm.
Observations omitted from a bootstrap training sample are called out-of-bag observations. Since about of original records are absent from a typical large bootstrap sample, they can be used to evaluate predictions from models that did not train on them. Out-of-bag error and bootstrap-based prediction-error corrections are related to, but distinct from, cross-validation. Their behavior depends on the learning algorithm, the evaluation loss, and the distinction between fitting and testing distributions.
When the target is uncertainty in a complete learning pipeline, data-dependent preprocessing, feature selection, and tuning ordinarily belong inside the resampled procedure. Holding selected features or tuning choices fixed changes the target to one conditional on those choices. Resampling the full pipeline still does not guarantee valid inference after unstable model selection.
Failure modes and nonregular problems
The ordinary bootstrap can fail even when the dataset is large and the computation is exact. A central example is estimation of the endpoint of a uniform distribution. Suppose are drawn from the uniform distribution on , and . Every ordinary bootstrap maximum satisfies : empirical resampling cannot generate an observation beyond the largest observed value.
The problem is more specific than the absence of extrapolation. Assuming the observed maximum is unique,
Thus the centered bootstrap maximum has a persistent atom at zero. In contrast, the appropriately scaled sampling error of the original maximum has a continuous limiting distribution. The bootstrap therefore fails to reproduce the endpoint estimator’s limiting law. A correctly specified parametric bootstrap for a uniform model can behave differently because it generates new observations from a continuous fitted distribution.
Other nonregular problems arise at parameter boundaries, with weakly identified parameters, or when an estimator changes discontinuously as the data change. Threshold estimation, some change-point problems, and post-model-selection distributions can require specialized theory. Correct bootstrap inference cannot be inferred merely from the fact that the estimator is consistent.
Heavy tails create another difficulty. If a population mean has infinite variance, ordinary root- normal theory need not apply, and the usual bootstrap of the mean can fail. Even with finite variance, a few extreme observations can make finite-sample bootstrap distributions highly unstable. Every bootstrap population is supported on the observed sample, so rare but influential values not represented in that sample receive no probability under ordinary empirical resampling.
One alternative is the -out-of- bootstrap, which resamples observations, often with . Another is subsampling, which commonly computes the statistic on smaller subsets without replacement. Appropriate recentering and rescaling can restore validity in some nonregular settings. These methods introduce their own conditions, including the choice of subset size and, in many formulations, knowledge or estimation of the statistic’s convergence rate. Smaller resamples are not an automatic remedy.
Bootstrapping also cannot repair failures of data collection or statistical identification. Selection bias, unmeasured confounding, systematic measurement error, and an incorrect missing-data mechanism are not removed by repeatedly resampling the same records. In causal inference, bootstrap variation can quantify uncertainty in an estimator under specified assumptions, but it does not establish those assumptions.
These limitations distinguish uncertainty conditional on the observed information from uncertainty about the whole scientific or sampling process. A narrow bootstrap distribution may accurately reflect one fitted procedure while omitting important sources of uncertainty that the resampling design never represents.
Historical development and computational practice
Bradley Efron introduced the bootstrap in a 1979 paper titled Bootstrap Methods: Another Look at the Jackknife. Earlier resampling ideas included Maurice Quenouille’s bias-reduction work in 1949 and John Tukey’s development of the jackknife in the 1950s. The jackknife systematically deletes observations, whereas the bootstrap approximates sampling from an estimated population. Efron’s formulation made the latter principle a general computational approach to statistical inference.
Subsequent developments extended both theory and scope. Rubin introduced the Bayesian bootstrap in 1981. Efron developed the BCa approach in 1987. Hans Künsch’s 1989 work helped establish block resampling for stationary dependent observations, and Dimitris Politis and Joseph Romano introduced the stationary bootstrap in 1994. Leo Breiman presented bagging in 1996. These developments addressed different problems: posterior uncertainty, interval accuracy, serial dependence, and predictive instability.
Bootstrap computation is usually an application of the Monte Carlo method. Conditional on the observed data, independent replicates estimate features of the bootstrap distribution. Simulation error generally decreases at a rate proportional to . Increasing fourfold therefore roughly halves Monte Carlo error for quantities to which that approximation applies; it does not halve the original estimator’s sampling error.
The required number of replicates depends on the output. Standard-error estimates often stabilize sooner than extreme quantiles. With , a nominal endpoint lies near the 25th ordered replicate, leaving relatively little information in that tail. For a fixed threshold whose true bootstrap exceedance probability is , the simulated exceedance fraction has standard error approximately
Quantile error additionally depends on the distribution’s density near the endpoint. More computation cannot compensate for an invalid resampling model.
Because replicates are largely independent, bootstrap calculations are well suited to parallel computing. Reproducibility requires explicit control of random-number streams, particularly when tasks run on different processors. Large-data implementations can use multinomial weights rather than physically copying rows. Methods such as the bag of little bootstraps combine calculations on smaller subsets with weighted resampling to approximate full-sample uncertainty, under conditions that justify the approximation.
Operational failures also require attention. A resample may contain only one outcome category, produce a singular design matrix, or cause a numerical optimizer not to converge. Discarding these replicates without accounting for their cause changes the simulated distribution. A computational specification therefore needs a defined statistic or failure rule for such datasets. Reporting the resampling unit, model restrictions, replicate count, interval construction, and handling of failed fits identifies which bootstrap procedure was actually performed.