{
  "id": "monte-carlo-dropout",
  "title": "Monte Carlo Dropout",
  "lang": "en",
  "source_lang": "en",
  "summary": "Monte Carlo dropout estimates neural-network predictive uncertainty by retaining dropout during inference and aggregating predictions from repeated stochastic forward passes.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "MC dropout",
    "Monte Carlo random dropout"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/monte-carlo-dropout/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/monte-carlo-dropout/",
      "json": "https://aiwiki.page/zh/monte-carlo-dropout.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Monte Carlo dropout is a method for estimating predictive uncertainty in an artificial neural network by keeping dropout active during inference and evaluating the same input repeatedly with different random dropout masks. The resulting predictions are combined to estimate a predictive mean and measures of variability. Under particular modeling assumptions, the procedure can be interpreted as approximate Bayesian inference rather than merely as a prediction-averaging technique."
    },
    {
      "heading": "Background and Bayesian interpretation",
      "text": "Dropout was introduced as a regularization technique for reducing overfitting. During training, it randomly suppresses selected activations or connections, so that the network cannot consistently rely on the same combinations of features. In ordinary inference, dropout is disabled and a deterministic network is used, with appropriate scaling to account for the training-time masking.\n\nMonte Carlo dropout instead retains random masking during inference. Each forward pass evaluates a different masked version of the trained network. These versions share learned parameters and are not independently trained models, but their predictions provide a computationally convenient approximation to an ensemble.\n\nThe modern Bayesian interpretation was developed by Yarin Gal and Zoubin Ghahramani in work published in 2016. Their analysis connected dropout training to variational inference in Bayesian models, including an interpretation in terms of approximate deep Gaussian processes.\n\nIn a Bayesian neural network, uncertainty about network parameters is represented by a posterior distribution. For training data \\(D\\), input \\(x\\), output \\(y\\), and parameters \\(W\\), the posterior predictive distribution is\n\n\\[p(y\\mid x,D) = \\int p(y\\mid x,W)\\,p(W\\mid D)\\,dW.\\]\n\nThis integral is generally intractable for large neural networks. The dropout interpretation replaces the posterior \\(p(W\\mid D)\\) with a restricted approximation \\(q_\\theta(W)\\), whose randomness is induced by dropout masks. Repeated masked forward passes then approximate integration under this distribution using Monte Carlo sampling.\n\nThe Bayesian interpretation is conditional on the choice of likelihood, prior, regularization, dropout parameterization, and approximations used in deriving the training objective. Enabling dropout in an arbitrary trained network does not by itself establish that its predictions are accurate samples from a Bayesian posterior."
    },
    {
      "heading": "Sampling procedure",
      "text": "Suppose an activation vector is \\(h\\), and each element is retained independently with probability \\(r\\). Under the widely used inverted-dropout convention,\n\n\\[z_j\\sim\\operatorname{Bernoulli}(r), \\qquad \\widetilde h_j=\\frac{z_jh_j}{r}.\\]\n\nThe scaling preserves each activation’s conditional expected value. A retention probability \\(r\\) corresponds to a dropout probability \\(1-r\\).\n\nFor a trained network, Monte Carlo dropout uses the following procedure:\n\n- Keep the intended dropout layers active while leaving learned parameters fixed.\n- Evaluate an input \\(x\\) with \\(T\\) independently sampled sets of masks.\n- Store the output of each stochastic forward pass.\n- Aggregate the outputs into a prediction and one or more uncertainty estimates.\n\nWriting the prediction from pass \\(t\\) as \\(f_t(x)\\), the estimated predictive mean is\n\n\\[\\widehat\\mu(x)=\\frac{1}{T}\\sum_{t=1}^{T}f_t(x).\\]\n\nThe estimated covariance of the sampled network outputs is\n\n\\[\\widehat\\Sigma_f(x) = \\frac{1}{T-1} \\sum_{t=1}^{T} \\bigl(f_t(x)-\\widehat\\mu(x)\\bigr) \\bigl(f_t(x)-\\widehat\\mu(x)\\bigr)^\\top,\\]\n\nfor \\(T>1\\). For scalar outputs, this reduces to a sample variance.\n\nIncreasing \\(T\\) reduces sampling error in these estimates, but does not improve the underlying posterior approximation. For independent samples with finite variance, the standard error of the estimated mean decreases proportionally to \\(T^{-1/2}\\)."
    },
    {
      "heading": "Regression and predictive variance",
      "text": "In regression, variability between masked-network predictions is usually interpreted as an approximation to uncertainty about the learned model. It is not necessarily the full uncertainty of a future observation.\n\nFor example, suppose the observation model is\n\n\\[y\\mid x,W\\sim \\mathcal N\\bigl(f_W(x),\\sigma^2\\bigr).\\]\n\nThe law of total variance gives\n\n\\[\\operatorname{Var}(y\\mid x,D) = \\mathbb E_{W\\mid D} \\left[\\operatorname{Var}(y\\mid x,W)\\right] + \\operatorname{Var}_{W\\mid D} \\left[\\mathbb E(y\\mid x,W)\\right].\\]\n\nUnder the dropout approximation, the predictive variance can therefore be estimated as\n\n\\[\\widehat{\\operatorname{Var}}(y\\mid x,D) = \\sigma^2+\\widehat\\Sigma_f(x)\\]\n\nfor scalar outputs.\n\nIf the network predicts an input-dependent observation variance \\(v_t(x)\\) on each pass, a corresponding estimate is\n\n\\[\\widehat{\\operatorname{Var}}(y\\mid x,D) = \\frac{1}{T}\\sum_{t=1}^{T}v_t(x) + \\widehat\\Sigma_f(x).\\]\n\nThese two components distinguish observation noise from variation among sampled models. A network trained only to produce point predictions does not automatically supply an observation-noise model; the dispersion of its dropout outputs alone should not be described as total predictive uncertainty."
    },
    {
      "heading": "Classification and uncertainty measures",
      "text": "For classification, each forward pass produces a vector of class probabilities,\n\n\\[p_t(c\\mid x).\\]\n\nThe Monte Carlo estimate of the predictive probability is\n\n\\[\\overline p(c\\mid x) = \\frac{1}{T}\\sum_{t=1}^{T}p_t(c\\mid x).\\]\n\nProbabilities are averaged after the class-probability transformation. Applying a softmax to averaged logits generally produces a different result because softmax is nonlinear.\n\nSeveral uncertainty scores can be calculated from these predictions.\n\nPredictive entropy\n\nThe entropy of the averaged prediction is\n\n\\[H[\\overline p] = -\\sum_c\\overline p(c\\mid x) \\log\\overline p(c\\mid x).\\]\n\nIt is large when the overall predictive distribution is spread across classes. However, it does not distinguish uncertainty caused by disagreement among sampled models from uncertainty present within every individual prediction.\n\nExpected entropy\n\nThe average entropy of individual predictions is\n\n\\[\\frac{1}{T}\\sum_{t=1}^{T}H[p_t].\\]\n\nIf every sampled model assigns similarly diffuse probabilities, this quantity can be high even when there is little disagreement between models.\n\nMutual-information estimate\n\nThe difference\n\n\\[H[\\overline p] - \\frac{1}{T}\\sum_{t=1}^{T}H[p_t]\\]\n\nestimates the mutual information between the predicted label and the sampled model parameters under the approximate posterior. It is often used as a measure of model disagreement.\n\nFor example, sampled models that confidently predict different classes can produce high predictive entropy but low expected entropy. Their difference is then high. Conversely, models that all produce nearly the same uncertain prediction can have high values of both entropies but a small difference."
    },
    {
      "heading": "Types of uncertainty",
      "text": "Monte Carlo dropout is commonly discussed within uncertainty quantification, especially the distinction between:\n\n- Epistemic uncertainty: uncertainty associated with limited knowledge of model parameters or model structure.\n- Aleatoric uncertainty: variability in observations that remains conditional on the available inputs, such as measurement noise or ambiguity not resolved by those inputs.\n\nVariation across dropout masks is primarily intended to approximate epistemic uncertainty within the chosen model family. Aleatoric uncertainty requires an appropriate likelihood or output representation.\n\nThis distinction is model-dependent rather than absolute. Variability that appears irreducible with one set of inputs may become predictable when additional information is available. Likewise, a restricted dropout approximation may fail to reveal important uncertainty about the model."
    },
    {
      "heading": "Implementation considerations",
      "text": "Dropout must match the trained model\n\nThe network should ordinarily have been trained with the dropout mechanism used for stochastic inference. Adding dropout only after training changes the computation without learning parameters adapted to that change.\n\nDropout location, mask structure, and rate are consequential hyperparameters. Masking individual activations, entire feature channels, or recurrent connections defines different distributions over network computations.\n\nIn recurrent neural networks, independently changing masks at every time step is not equivalent to keeping a mask fixed across a sequence. Bayesian interpretations for recurrent dropout depend on the masking scheme.\n\nOther layers need separate treatment\n\nIn frameworks such as PyTorch and TensorFlow, training and evaluation modes can affect more than dropout. Setting an entire model to training mode can also change batch normalization behavior, including the use or updating of batch statistics.\n\nMonte Carlo dropout implementations therefore commonly activate selected dropout modules while keeping batch-normalization layers in evaluation mode. Recording gradients is unnecessary for ordinary uncertainty estimation.\n\nSampling and computational cost\n\nRepeatedly resetting the random generator to the same state can reproduce identical masks and defeat the sampling procedure. Independent masks are needed across Monte Carlo passes.\n\nA single shared mask can nevertheless be intentional when evaluating multiple inputs as one sampled function. This matters for estimating correlations between predictions at different inputs.\n\nUsing \\(T\\) stochastic passes costs approximately \\(T\\) forward evaluations, although parallel execution or batching can reduce elapsed time. Memory use can be reduced by accumulating means and variances rather than storing every prediction. More samples improve the precision of these estimates but do not remedy model misspecification."
    },
    {
      "heading": "Applications and comparison with alternatives",
      "text": "Monte Carlo dropout has been used in regression, computer vision, semantic segmentation, and other prediction tasks where a confidence score alone is insufficient. In dense prediction, it can produce spatial maps of disagreement across stochastic outputs.\n\nIt is also used in active learning, where uncertainty measures help select unlabeled examples for annotation. Disagreement-based criteria aim to identify examples that may provide information about model parameters, although their effectiveness depends on the quality of the uncertainty approximation.\n\nThe method resembles ensemble learning, but differs from ensembles of independently trained networks. Dropout samples share a common set of learned parameters and explore a restricted family of subnetworks. Independently trained ensembles can represent different solutions found during optimization, at the cost of training and storing multiple models.\n\nCompared with more elaborate Bayesian inference methods, Monte Carlo dropout is relatively straightforward to add to an existing dropout-trained model. Its simplicity is accompanied by limited flexibility in the approximate posterior."
    },
    {
      "heading": "Limitations and evaluation",
      "text": "Monte Carlo dropout does not guarantee reliable probability calibration. Prediction dispersion can be too small or too large, and a low dropout variance is not proof that a prediction is correct.\n\nIts principal limitations include:\n\n- Restricted posterior representation. Bernoulli masking explores only a limited family of parameter configurations and may miss important modes or dependencies.\n- Sensitivity to design choices. Dropout rate, layer placement, training objective, and regularization affect both accuracy and estimated uncertainty.\n- Unreliable behavior under distribution shift. Inputs unlike the training data do not necessarily produce large disagreement. The method is not a guaranteed detector of distribution shift or anomalous inputs.\n- Inference overhead. Many stochastic passes can be expensive for large networks or latency-sensitive applications.\n- Incomplete noise modeling. Dispersion across predictions does not, by itself, account for observation noise.\n- Approximate predictive distributions. Intervals derived from a Gaussian approximation may be misleading when the sampled predictive distribution is skewed or multimodal.\n\nEvaluation therefore distinguishes predictive accuracy, calibration, uncertainty quality, and computational cost. Relevant measurements include predictive log-likelihood, proper scoring rules, empirical interval coverage, and performance when predictions with high estimated uncertainty are withheld. Dropout settings and sampling budgets can be assessed using a validation set, with final evaluation performed on a separate test set."
    }
  ],
  "links": [
    "artificial-neural-network",
    "dropout",
    "regularization",
    "overfitting",
    "neural-network-inference",
    "variational-inference",
    "monte-carlo-method",
    "expected-value",
    "variance",
    "multiclass-classification",
    "information-entropy",
    "hyperparameter",
    "recurrent-neural-network",
    "pytorch",
    "tensorflow",
    "batch-normalization",
    "parallel-computing",
    "computer-vision",
    "semantic-segmentation",
    "active-learning",
    "ensemble-learning",
    "probability-calibration",
    "training-data",
    "distribution-shift",
    "validation-set"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}