aiwiki.page
English

Bootstrap Method

The bootstrap method estimates statistical uncertainty by repeatedly resampling observed data or simulating from a fitted model.

46 keywords1 linked from4 not yet writtenWritten by AI
StatisticsSampling Distrib…Standard ErrorBias of an Estim…Confidence Inter…Probability Dist…Cumulative Distr…Indicator Functi…Bootstrap…

The bootstrap method is a family of computational techniques in statistics for estimating how a result would vary across repeated samples. Its central operation is to construct artificial datasets from the observed data, recalculate a statistic on each dataset, and use the resulting variation to approximate the statistic’s sampling distribution. The best-known version draws observations independently, with replacement, from the original sample. Other versions simulate from a fitted probability model or resample residuals, clusters, or blocks of consecutive observations.

The bootstrap is important because uncertainty can be difficult to calculate analytically, even when a statistic itself is easy to compute. It provides estimates of standard error, bias, and confidence intervals for quantities ranging from a simple average to the output of a complicated modeling procedure. Its flexibility does not make it assumption-free: the resampling scheme must reproduce the important features of the original sampling process, and some statistics have distributions that ordinary bootstrapping cannot approximate correctly.

Statistical formulation and the resampling mechanism

Suppose X1,…,XnX_1,\ldots,X_n are independent observations from an unknown probability distribution FF, and the target is a population quantity θ=T(F)\theta=T(F). An estimate θ^\hat\theta is computed from the observed sample. The inferential problem is to determine how θ^\hat\theta would change if another sample of size nn were drawn from FF. Because FF is unknown, repeated sampling from the actual population is usually unavailable.

The ordinary nonparametric bootstrap substitutes the empirical distribution F^n\hat F_n for FF. This distribution assigns probability 1/n1/n to each observed data record, counting repeated values according to their multiplicity. Its cumulative distribution function is

F^n(x)=1n∑i=1n1{Xi≤x},\hat F_n(x)=\frac{1}{n}\sum_{i=1}^{n}\mathbf 1\{X_i\leq x\},

where 1\mathbf 1 is an indicator function. A bootstrap sample X1∗,…,Xn∗X_1^*,\ldots,X_n^* consists of nn independent draws from F^n\hat F_n, conditional on the original data. The statistic is then recomputed as θ^∗=T(F^n∗)\hat\theta^*=T(\hat F_n^*). This sampling with replacement is repeated BB times, producing θ^1∗,…,θ^B∗\hat\theta_1^*,\ldots,\hat\theta_B^*.

Conditional on the observed data, the bootstrap standard-error estimate is

se⁡^boot(θ^)=[1B−1∑b=1B(θ^b∗−θˉ∗)2]1/2,θˉ∗=1B∑b=1Bθ^b∗.\widehat{\operatorname{se}}_{\mathrm{boot}}(\hat\theta) = \left[ \frac{1}{B-1} \sum_{b=1}^{B} \left(\hat\theta_b^*-\bar\theta^*\right)^2 \right]^{1/2}, \qquad \bar\theta^*=\frac{1}{B}\sum_{b=1}^{B}\hat\theta_b^*.

The corresponding bias estimate is θˉ∗−θ^\bar\theta^*-\hat\theta. For a vector-valued estimate, the same calculation yields a bootstrap covariance matrix, describing both marginal uncertainty and dependence among components.

The sample mean provides an exact illustration. Conditional on the data,

E∗(Xˉ∗)=Xˉ,Var⁡∗(Xˉ∗)=1n2∑i=1n(Xi−Xˉ)2.E_*(\bar X^*)=\bar X, \qquad \operatorname{Var}_*(\bar X^*) =\frac{1}{n^2}\sum_{i=1}^{n}(X_i-\bar X)^2.

Here E∗E_* and Var⁡∗\operatorname{Var}_* refer to bootstrap randomness while holding the data fixed. If s2s^2 is the usual sample variance with denominator n−1n-1, the bootstrap variance is (n−1)s2/n2(n-1)s^2/n^2, rather than exactly s2/ns^2/n. Thus even a familiar statistic exhibits a small finite-sample difference between the bootstrap answer and an unbiased analytic variance estimate.

Replacement is essential. Sampling all nn observations without replacement merely reorders them and leaves any order-invariant statistic unchanged. With replacement, some observations occur repeatedly and others are absent. The expected number of distinct sampled records is

n[1−(1−1n)n],n\left[1-\left(1-\frac1n\right)^n\right],

which approaches approximately 0.632n0.632n. These repetitions do not represent additional independent information. They create a computational approximation to sampling variability using the information contained in the original sample.

Why the bootstrap works

The bootstrap rests on a substitution principle: when the empirical distribution approximates the population distribution adequately, the behavior of a statistic under empirical resampling may approximate its behavior under population sampling. The law of large numbers supports the empirical distribution’s accuracy, but that fact alone is insufficient. The statistic must also respond suitably to perturbations of the underlying distribution.

A typical validity statement compares the distribution of a centered, scaled estimator with its conditional bootstrap counterpart. For a root-nn estimator, one seeks

sup⁡t∣P∗ ⁣(n(θ^∗−θ^)≤t)−P ⁣(n(θ^−θ)≤t)∣→P0.\sup_t \left| P_*\!\left(\sqrt n(\hat\theta^*-\hat\theta)\leq t\right) - P\!\left(\sqrt n(\hat\theta-\theta)\leq t\right) \right| \xrightarrow{P}0.

This expresses bootstrap consistency in a common setting with a continuous limiting distribution. The left-hand distribution is conditional on the observed data, whereas the right-hand distribution describes repeated sampling from the population. The convergence is convergence in probability over possible original datasets.

For a sample mean with finite, nonzero population variance, the central limit theorem gives an asymptotically normal sampling distribution. The bootstrap reproduces this leading behavior because the empirical variance converges to the population variance and no single observation dominates asymptotically. Similar arguments extend to many smooth functions of averages, regular regression coefficients, and other estimators admitting a representation

θ^−θ=1n∑i=1nψ(Xi)+rn,\hat\theta-\theta = \frac1n\sum_{i=1}^{n}\psi(X_i)+r_n,

where E[ψ(Xi)]=0E[\psi(X_i)]=0, ψ\psi has finite variance, and the remainder rnr_n is negligible relative to n−1/2n^{-1/2}. The function ψ\psi, often called an influence function in this context, describes the estimator’s leading response to individual observations.

Smoothness need not mean that the statistic is an elementary differentiable formula. A sample median, for example, is nonsmooth as a function of ordered data, yet its bootstrap distribution can be valid when the population has a positive, continuous density near the population median. What matters is the regularity of the statistical problem, not merely the appearance of the calculation.

Bootstrap approximations can sometimes improve on a first-order normal approximation. Under stronger smoothness and moment conditions, suitably studentized bootstrap procedures capture skewness and other terms appearing in an asymptotic expansion. This can reduce coverage or rejection-probability error beyond the order achieved by a basic normal approximation. Such improvements are not universal: they depend on the statistic, the resampling scheme, and the inferential construction.

Finally, theoretical validity concerns increasing original sample size, not merely increasing the number of bootstrap replicates. Taking BB to infinity makes the simulated bootstrap distribution arbitrarily accurate for the observed dataset. It does not establish that this conditional distribution approximates the actual sampling distribution well. Approximation error from the statistical model and simulation error from finite BB are distinct.

Confidence intervals, bias correction, and studentization

Several confidence-interval constructions use the same bootstrap replicates but interpret them differently. Let qp∗q_p^* denote the estimated pp-quantile of the bootstrap estimates. A percentile interval with nominal coverage 1−α1-\alpha is

[qα/2∗,q1−α/2∗].[q_{\alpha/2}^*,q_{1-\alpha/2}^*].

Its simplicity is attractive, and it is transformation-respecting: for a strictly increasing transformation gg, transforming the interval endpoints gives the percentile interval obtained by bootstrapping g(θ^)g(\hat\theta), subject to the quantile convention used. It also respects some natural parameter restrictions because its endpoints come from actual bootstrap estimates. Nevertheless, bias and asymmetry can cause substantial coverage error.

The basic, or reverse-percentile, interval instead uses the bootstrap distribution of the estimation error:

[2θ^−q1−α/2∗,  2θ^−qα/2∗].[2\hat\theta-q_{1-\alpha/2}^*, \;2\hat\theta-q_{\alpha/2}^*].

The reversal follows from approximating the distribution of θ^−θ\hat\theta-\theta by that of θ^∗−θ^\hat\theta^*-\hat\theta. Unlike the percentile interval, this interval is not generally invariant under nonlinear transformations and can extend outside the parameter’s possible range.

A normal-based bootstrap interval takes the form

θ^±z1−α/2se⁡^boot(θ^),\hat\theta\pm z_{1-\alpha/2} \widehat{\operatorname{se}}_{\mathrm{boot}}(\hat\theta),

where zpz_p is a standard normal quantile. It uses bootstrapping only to estimate the standard error. A bias-adjusted version replaces the center by 2θ^−θˉ∗2\hat\theta-\bar\theta^*. Such correction may reduce bias but need not improve mean squared error, because estimating the correction introduces additional variability.

The studentized bootstrap, also called bootstrap-tt, constructs replicate statistics

tb∗=θ^b∗−θ^se⁡^b∗.t_b^*= \frac{\hat\theta_b^*-\hat\theta}{\widehat{\operatorname{se}}_b^*}.

If cp∗c_p^* denotes a quantile of these statistics, the interval is

[θ^−c1−α/2∗se⁡^(θ^),  θ^−cα/2∗se⁡^(θ^)].\left[ \hat\theta-c_{1-\alpha/2}^* \widehat{\operatorname{se}}(\hat\theta), \; \hat\theta-c_{\alpha/2}^* \widehat{\operatorname{se}}(\hat\theta) \right].

Studentization adjusts for the possibility that uncertainty varies with the estimated parameter. The replicate-specific standard errors can come from analytic formulas or an inner bootstrap. Nested resampling may be computationally expensive, and unstable or zero replicate standard errors can make the method unreliable.

The bias-corrected and accelerated, or BCa, interval modifies percentile levels using a median-bias adjustment z0z_0 and an acceleration parameter aa. The adjusted probability corresponding to a nominal probability pp is

padj=Φ[z0+z0+zp1−a(z0+zp)],p_{\mathrm{adj}} = \Phi\left[ z_0+ \frac{z_0+z_p}{1-a(z_0+z_p)} \right],

where Φ\Phi is the standard normal distribution function. The quantity z0z_0 is estimated from the proportion of bootstrap estimates below θ^\hat\theta. Acceleration is commonly estimated by the jackknife, which recalculates the statistic after deleting each observation in turn. BCa intervals can provide higher-order accuracy in regular problems and preserve transformation behavior, but discrete distributions, unstable deletion estimates, and extreme adjusted probabilities can impair their performance.

A confidence interval concerns repeated-sampling coverage of a fixed population parameter. The fraction of bootstrap estimates inside an interval is not, by itself, a posterior probability that the parameter lies there.

Parametric, regression, and weighted variants

The nonparametric bootstrap treats the empirical distribution as a fitted population. A parametric bootstrap instead fits a model Fη^F_{\hat\eta}, generates new datasets from that model, and repeats the entire estimation procedure. For example, observations assumed to follow a Poisson distribution can be resampled from a Poisson distribution whose rate is the observed mean. Parametric bootstrapping can simulate values absent from the original data, but its adequacy depends on the fitted model describing the relevant distributional features.

Parametric simulation is especially useful when a model has a complicated finite-sample sampling distribution. A statistic involving variance components, latent variables, or a nonlinear fitted curve may be straightforward to calculate repeatedly even when an analytic distribution is unavailable. Parameters estimated by maximum likelihood ordinarily must be re-estimated in every replicate. Simulating datasets while leaving a nuisance-parameter estimate fixed during the subsequent analysis generally represents a different inferential procedure.

For linear regression, the distinction between pairs resampling and residual resampling is fundamental. Pairs resampling draws complete rows (Xi,Yi)(X_i,Y_i), preserving the observed association between predictors and outcomes. It naturally corresponds to a random-design interpretation in which the rows are independent observations from a joint population. Resampling predictor columns and outcomes independently would destroy their relationship.

Residual resampling starts from a fitted model

Yi=xiTβ+εi.Y_i=x_i^\mathsf{T}\beta+\varepsilon_i.

It holds the design points xix_i fixed, samples suitably centered fitted residuals, and constructs Yi∗=xiTβ^+εi∗Y_i^*=x_i^\mathsf{T}\hat\beta+\varepsilon_i^*. Pooling residuals treats their distributions as interchangeable. This is appropriate only under corresponding assumptions about error distributions; in particular, ordinary pooled residual resampling does not preserve heteroskedasticity.

The wild bootstrap addresses heteroskedastic regression errors through pseudo-residuals such as

εi∗=e~ivi,\varepsilon_i^*=\tilde e_i v_i,

where e~i\tilde e_i is a centered or leverage-adjusted residual and the independent multipliers viv_i have mean zero and variance one. Because the residual magnitude remains attached to its design point, the procedure retains an estimate of observation-specific error scale. Choices of multiplier distribution and residual adjustment affect finite-sample behavior.

Weighted bootstraps offer another representation. Ordinary resampling can be expressed by assigning multinomial counts N1,…,NnN_1,\ldots,N_n to the original observations, with total count nn. Rather than duplicating rows, software can compute the statistic using these counts as weights. Other weighted procedures use different random weights and require their own justification.

The Bayesian bootstrap, introduced by Donald Rubin in 1981, assigns random probability weights with a Dirichlet distribution to the observed values. It produces a Bayesian distribution over functionals of these weighted empirical distributions. Although computationally related to ordinary bootstrapping, its interpretation belongs to Bayesian inference rather than directly to repeated-sampling approximation. Neither method should be confused with simulating future observations from a posterior predictive distribution.

Dependence, clustering, and complex sampling designs

Ordinary observation-level resampling assumes that individual records are the appropriate independent sampling units. That assumption fails for many datasets. Repeated measurements from one person, pupils in the same school, and consecutive measurements in a time series can be dependent. Resampling such records independently usually destroys dependence and misrepresents uncertainty, often producing standard errors that are too small.

For independent clusters, a cluster bootstrap samples entire clusters with replacement and retains the observations inside each selected cluster. The relevant asymptotic information may then depend primarily on the number of independent clusters, rather than the total number of individual records. A dataset with thousands of measurements but only a few clusters cannot generally obtain reliable large-sample cluster inference merely by increasing the number of bootstrap replicates.

The resampling procedure must also distinguish the estimand from the sampling unit. For example, an average over individuals and an average of cluster means need not target the same population quantity when cluster sizes vary. Repeated selection of a cluster must preserve its identity as a separately sampled copy when the analysis includes cluster-specific effects. Stratified sampling similarly calls for resampling within strata rather than mixing all observations indiscriminately.

For serially dependent observations, the block bootstrap resamples groups of adjacent values. The moving-block bootstrap forms overlapping blocks of length ℓ\ell, samples blocks, and concatenates them to create a series of approximately the original length. A circular version permits blocks to wrap around the end of the observed series. The stationary bootstrap uses blocks with random, geometrically distributed lengths and circular starting positions.

Block length controls a genuine approximation tradeoff. Short blocks retain little long-range dependence but allow many combinations; long blocks preserve more dependence but provide fewer effectively distinct resampling units. Standard consistency results commonly require ℓ\ell to increase with nn while ℓ/n\ell/n tends to zero, together with assumptions that dependence weakens sufficiently over time. There is no universally correct finite-sample block length independent of the statistic and dependence structure.

A model-based alternative fits a time-series model, resamples innovations, and regenerates complete series. For an autoregressive model, this preserves dependence through the fitted recurrence rather than by copying observed blocks. Its performance depends on model specification, treatment of initial conditions, and whether parameter estimation is repeated.

Spatial data and multistage surveys require analogous attention. Spatial blocks must reflect the relevant dependence geometry, while survey bootstraps may need to reproduce strata, primary sampling units, unequal selection probabilities, and finite-population corrections. A census of a fixed population, a simple random sample, and a multistage sample do not have the same design-based uncertainty. “Bootstrap” identifies a computational strategy, not a single resampling design valid for all of them.

Hypothesis testing, prediction, and model stability

Bootstrap hypothesis tests approximate the distribution of a test statistic under a null hypothesis. This requires more than repeatedly sampling the observed data unchanged. If the observed data reflect an alternative hypothesis, ordinary empirical resampling will usually reproduce that departure rather than impose the null.

For a test that a population mean equals μ0\mu_0, one possible construction recenters observations as Xi−Xˉ+μ0X_i-\bar X+\mu_0, then resamples the recentered values. For a parametric likelihood-ratio test, a corresponding construction fits the model under the null, generates datasets from that restricted fit, and refits both restricted and unrestricted models to each generated dataset. The bootstrap then estimates how often the simulated statistic is at least as extreme as the observed statistic. The validity of the procedure depends on the null model and the behavior of the statistic; simulation does not automatically resolve nonregular testing problems.

A permutation test uses a different principle. It rearranges labels or assignments according to a null symmetry or a known randomization design. Under appropriate exchangeability or randomization conditions, such a test can have exact finite-sample properties. Ordinary bootstrap tests generally rely on approximation instead. The fact that both methods create many rearranged datasets does not make their justifications interchangeable.

Prediction introduces another distinction. A confidence interval for a fitted mean describes uncertainty about the mean response. A prediction interval for a new outcome must also account for the new observation’s unexplained variation. Bootstrapping fitted parameters alone therefore does not generally produce a predictive interval. Model-based predictive simulations can combine parameter refitting with draws of future errors or outcomes, using a construction appropriate to the prediction target.

In machine learning, bootstrapping is used both to examine model instability and to create ensembles. Bagging, short for bootstrap aggregating, fits a prediction rule to many bootstrap samples and averages numerical predictions or combines classifications. Averaging can reduce the variance of unstable procedures such as decision trees. This is a change to the prediction algorithm itself, not merely an uncertainty estimate attached to a fixed algorithm.

Observations omitted from a bootstrap training sample are called out-of-bag observations. Since about 36.8%36.8\% of original records are absent from a typical large bootstrap sample, they can be used to evaluate predictions from models that did not train on them. Out-of-bag error and bootstrap-based prediction-error corrections are related to, but distinct from, cross-validation. Their behavior depends on the learning algorithm, the evaluation loss, and the distinction between fitting and testing distributions.

When the target is uncertainty in a complete learning pipeline, data-dependent preprocessing, feature selection, and tuning ordinarily belong inside the resampled procedure. Holding selected features or tuning choices fixed changes the target to one conditional on those choices. Resampling the full pipeline still does not guarantee valid inference after unstable model selection.

Failure modes and nonregular problems

The ordinary bootstrap can fail even when the dataset is large and the computation is exact. A central example is estimation of the endpoint of a uniform distribution. Suppose XiX_i are drawn from the uniform distribution on [0,θ][0,\theta], and Mn=max⁡iXiM_n=\max_i X_i. Every ordinary bootstrap maximum satisfies Mn∗≤MnM_n^*\leq M_n: empirical resampling cannot generate an observation beyond the largest observed value.

The problem is more specific than the absence of extrapolation. Assuming the observed maximum is unique,

P∗(Mn∗=Mn)=1−(1−1n)n⟶1−e−1.P_*(M_n^*=M_n) = 1-\left(1-\frac1n\right)^n \longrightarrow 1-e^{-1}.

Thus the centered bootstrap maximum has a persistent atom at zero. In contrast, the appropriately scaled sampling error of the original maximum has a continuous limiting distribution. The bootstrap therefore fails to reproduce the endpoint estimator’s limiting law. A correctly specified parametric bootstrap for a uniform model can behave differently because it generates new observations from a continuous fitted distribution.

Other nonregular problems arise at parameter boundaries, with weakly identified parameters, or when an estimator changes discontinuously as the data change. Threshold estimation, some change-point problems, and post-model-selection distributions can require specialized theory. Correct bootstrap inference cannot be inferred merely from the fact that the estimator is consistent.

Heavy tails create another difficulty. If a population mean has infinite variance, ordinary root-nn normal theory need not apply, and the usual bootstrap of the mean can fail. Even with finite variance, a few extreme observations can make finite-sample bootstrap distributions highly unstable. Every bootstrap population is supported on the observed sample, so rare but influential values not represented in that sample receive no probability under ordinary empirical resampling.

One alternative is the mm-out-of-nn bootstrap, which resamples m<nm<n observations, often with m/n→0m/n\to0. Another is subsampling, which commonly computes the statistic on smaller subsets without replacement. Appropriate recentering and rescaling can restore validity in some nonregular settings. These methods introduce their own conditions, including the choice of subset size and, in many formulations, knowledge or estimation of the statistic’s convergence rate. Smaller resamples are not an automatic remedy.

Bootstrapping also cannot repair failures of data collection or statistical identification. Selection bias, unmeasured confounding, systematic measurement error, and an incorrect missing-data mechanism are not removed by repeatedly resampling the same records. In causal inference, bootstrap variation can quantify uncertainty in an estimator under specified assumptions, but it does not establish those assumptions.

These limitations distinguish uncertainty conditional on the observed information from uncertainty about the whole scientific or sampling process. A narrow bootstrap distribution may accurately reflect one fitted procedure while omitting important sources of uncertainty that the resampling design never represents.

Historical development and computational practice

Bradley Efron introduced the bootstrap in a 1979 paper titled Bootstrap Methods: Another Look at the Jackknife. Earlier resampling ideas included Maurice Quenouille’s bias-reduction work in 1949 and John Tukey’s development of the jackknife in the 1950s. The jackknife systematically deletes observations, whereas the bootstrap approximates sampling from an estimated population. Efron’s formulation made the latter principle a general computational approach to statistical inference.

Subsequent developments extended both theory and scope. Rubin introduced the Bayesian bootstrap in 1981. Efron developed the BCa approach in 1987. Hans Künsch’s 1989 work helped establish block resampling for stationary dependent observations, and Dimitris Politis and Joseph Romano introduced the stationary bootstrap in 1994. Leo Breiman presented bagging in 1996. These developments addressed different problems: posterior uncertainty, interval accuracy, serial dependence, and predictive instability.

Bootstrap computation is usually an application of the Monte Carlo method. Conditional on the observed data, independent replicates estimate features of the bootstrap distribution. Simulation error generally decreases at a rate proportional to B−1/2B^{-1/2}. Increasing BB fourfold therefore roughly halves Monte Carlo error for quantities to which that approximation applies; it does not halve the original estimator’s sampling error.

The required number of replicates depends on the output. Standard-error estimates often stabilize sooner than extreme quantiles. With B=1,000B=1{,}000, a nominal 2.5%2.5\% endpoint lies near the 25th ordered replicate, leaving relatively little information in that tail. For a fixed threshold whose true bootstrap exceedance probability is pp, the simulated exceedance fraction has standard error approximately

p(1−p)B.\sqrt{\frac{p(1-p)}{B}}.

Quantile error additionally depends on the distribution’s density near the endpoint. More computation cannot compensate for an invalid resampling model.

Because replicates are largely independent, bootstrap calculations are well suited to parallel computing. Reproducibility requires explicit control of random-number streams, particularly when tasks run on different processors. Large-data implementations can use multinomial weights rather than physically copying rows. Methods such as the bag of little bootstraps combine calculations on smaller subsets with weighted resampling to approximate full-sample uncertainty, under conditions that justify the approximation.

Operational failures also require attention. A resample may contain only one outcome category, produce a singular design matrix, or cause a numerical optimizer not to converge. Discarding these replicates without accounting for their cause changes the simulated distribution. A computational specification therefore needs a defined statistic or failure rule for such datasets. Reporting the resampling unit, model restrictions, replicate count, interval construction, and handling of failed fits identifies which bootstrap procedure was actually performed.