{
  "id": "thompson-sampling",
  "title": "Thompson Sampling",
  "lang": "en",
  "source_lang": "en",
  "summary": "Thompson sampling is a Bayesian decision-making method that balances exploration and exploitation by selecting actions optimal under randomly sampled models of uncertain rewards.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Posterior sampling",
    "Probability matching"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/thompson-sampling/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/thompson-sampling/",
      "json": "https://aiwiki.page/zh/thompson-sampling.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Thompson sampling is a randomized algorithm for sequential decision-making under uncertainty. At each step, it samples a possible model of the environment from its current posterior distribution, chooses an action that is optimal under that sampled model, observes the outcome, and updates its beliefs. It is best known as a solution to the multi-armed bandit problem, where a decision-maker repeatedly chooses among alternatives with initially unknown rewards. Its central purpose is to balance learning about uncertain alternatives with using alternatives that already appear valuable."
    },
    {
      "heading": "Decision problem",
      "text": "In the standard stochastic bandit setting, there are \\(K\\) available actions, traditionally called arms. Choosing action \\(a\\) produces a reward drawn from an unknown, fixed probability distribution with mean \\(\\mu_a\\). Only the reward of the chosen action is observed; the rewards that other actions would have produced remain unknown.\n\nThe decision-maker seeks to accumulate reward over a sequence of rounds. This creates the exploration–exploitation trade-off:\n\n- Exploration selects actions partly to learn about their rewards.\n- Exploitation selects actions that currently appear to offer high rewards.\n\nAlways choosing the action with the highest estimated mean can prematurely abandon a good alternative after unlucky observations. Exploring indefinitely at a fixed rate, however, can waste many decisions on clearly inferior alternatives.\n\nThompson sampling addresses this tension through uncertainty in the reward model. An uncertain action can be selected when a posterior sample makes it appear best, while an action that is consistently supported by observations tends to win more frequently."
    },
    {
      "heading": "Bayesian formulation",
      "text": "Thompson sampling uses Bayesian inference to represent uncertainty about the environment. Let \\(\\theta\\) denote the unknown model parameters and let \\(H_t\\) contain the actions and observations available before round \\(t\\). The posterior distribution is\n\n\\[p(\\theta\\mid H_t).\\]\n\nLet \\(m(a,\\theta)\\) denote the expected reward of action \\(a\\) under parameters \\(\\theta\\). A round of Thompson sampling proceeds as follows:\n\n- Draw a parameter sample:\n\n\\[\\widetilde{\\theta}_t\\sim p(\\theta\\mid H_t).\\]\n- Choose an action that maximizes expected reward in the sampled model:\n\n\\[A_t\\in\\operatorname*{arg\\,max}_{a}m(a,\\widetilde{\\theta}_t).\\]\n- Observe reward \\(R_t\\).\n- Update the posterior using the likelihood of the observation.\n\nThe update takes the form\n\n\\[p(\\theta\\mid H_{t+1}) \\propto p(R_t\\mid A_t,\\theta)\\,p(\\theta\\mid H_t).\\]\n\nThe sampled quantity is a model parameter, or an expected reward implied by that parameter—not normally a simulated realization of the next reward. Sampling a noisy reward and choosing the largest such realization is a different procedure.\n\nFor independent arm models, implementations often sample one parameter for each arm. If actions share parameters or have correlated uncertainties, sampling must preserve that joint structure.\n\nProbability matching\n\nIf the optimal action is unique with posterior probability one, exact Thompson sampling satisfies\n\n\\[\\Pr(A_t=a\\mid H_t) = \\Pr\\!\\left( a=\\operatorname*{arg\\,max}_{b}m(b,\\theta) \\mid H_t \\right).\\]\n\nThus, each action is chosen with the posterior probability that it is optimal. This property explains the alternative name probability matching. If ties have positive probability, the equality requires a consistent tie-breaking rule.\n\nAn action’s probability of being optimal is not the same as its posterior mean reward. An uncertain action may have a lower mean estimate but still have a substantial chance of being the best."
    },
    {
      "heading": "Beta–Bernoulli example",
      "text": "A common example involves binary rewards, such as a click or no click. For action \\(a\\), suppose\n\n\\[R_t\\mid A_t=a,p_a\\sim\\operatorname{Bernoulli}(p_a),\\]\n\nwhere \\(p_a\\) is its unknown success probability.\n\nAssign each \\(p_a\\) an independent beta distribution:\n\n\\[p_a\\sim\\operatorname{Beta}(\\alpha_a,\\beta_a), \\qquad \\alpha_a,\\beta_a>0.\\]\n\nAfter observing \\(s_a\\) successes and \\(f_a\\) failures for that action, its posterior is\n\n\\[p_a\\mid H_t \\sim \\operatorname{Beta}(\\alpha_a+s_a,\\beta_a+f_a).\\]\n\nThe beta distribution is a conjugate prior for the Bernoulli likelihood, so the posterior update requires only two parameters per action.\n\nThe resulting procedure is:\n\nInitialize alpha[a] > 0 and beta[a] > 0 for every action a\n\nFor each round:\n    For every action a:\n        Sample q[a] from Beta(alpha[a], beta[a])\n\n    Choose an action with the largest q[a]\n    Observe binary reward r\n\n    alpha[chosen] += r\n    beta[chosen]  += 1 - r\n\nA \\(\\operatorname{Beta}(1,1)\\) prior is uniform over success probabilities from zero to one. Other parameter choices express different prior beliefs and strengths of evidence.\n\nFor example, an extensively observed action with a success rate near \\(0.60\\) may have a narrow posterior, while a little-tested action has a much wider posterior. The uncertain action sometimes produces a higher sampled success probability and is selected. Further observations then determine whether its apparent promise is supported.\n\nWith \\(K\\) independent arms, this basic implementation uses \\(O(K)\\) work per decision and \\(O(K)\\) storage, assuming constant-cost beta sampling."
    },
    {
      "heading": "Exploration and performance",
      "text": "Thompson sampling explores through posterior uncertainty rather than through a separate instruction to take random actions. As evidence accumulates against an action, the posterior probability that it is optimal typically decreases, reducing its selection frequency.\n\nThis is not equivalent to explicitly calculating the value of information. Thompson sampling does not generally solve the full Bayesian planning problem of selecting actions to maximize expected reward over all future rounds. Instead, it acts optimally in one sampled model. It can perform well without being Bayes-optimal for a particular finite horizon.\n\nRegret\n\nPerformance is often measured using regret. In a stationary bandit, let\n\n\\[\\mu^*=\\max_a\\mu_a.\\]\n\nThe expected cumulative pseudo-regret after \\(T\\) rounds is\n\n\\[\\mathcal{R}_T = \\mathbb{E}\\!\\left[ \\sum_{t=1}^{T}(\\mu^*-\\mu_{A_t}) \\right].\\]\n\nEquivalently, with \\(\\Delta_a=\\mu^*-\\mu_a\\) and \\(N_a(T)\\) denoting the number of times action \\(a\\) is selected,\n\n\\[\\mathcal{R}_T = \\sum_{a:\\Delta_a>0} \\Delta_a\\,\\mathbb{E}[N_a(T)].\\]\n\nTwo common perspectives are:\n\n- Frequentist regret, which fixes the true environment and averages over reward noise and algorithmic randomization.\n- Bayesian regret, which additionally averages over an assumed prior on environments.\n\nFor standard finite-armed Bernoulli bandits with independent beta priors, Thompson sampling has logarithmic-in-\\(T\\), problem-dependent regret guarantees. Suitable versions attain asymptotically optimal leading behavior under established assumptions. The constants depend on how difficult it is to distinguish inferior actions from the best action.\n\nThese results do not provide a universal guarantee for every application called Thompson sampling. The reward model, prior, action space, and accuracy of posterior sampling matter."
    },
    {
      "heading": "Extensions",
      "text": "Contextual bandits\n\nIn a contextual bandit, the decision-maker observes information \\(x_t\\) before choosing an action. Expected rewards depend on both the action and the context:\n\n\\[m(a,x_t,\\theta).\\]\n\nThompson sampling draws \\(\\widetilde{\\theta}_t\\) and chooses\n\n\\[A_t\\in\\operatorname*{arg\\,max}_a m(a,x_t,\\widetilde{\\theta}_t).\\]\n\nExamples include linear reward models, logistic models for binary outcomes, and models using neural networks. Shared parameters allow an observation about one action or context to inform predictions elsewhere.\n\nThe tractability of inference varies substantially. Linear models with suitable Gaussian assumptions permit efficient exact updates; more complex models often require approximate inference.\n\nStructured and combinatorial actions\n\nAn action may be an assortment of products, a route through a network, or another structured object rather than one item from a short list. Thompson sampling then draws uncertain parameters and solves a corresponding optimization problem.\n\nThis separates statistical inference from action selection: a solver for a known-parameter problem can sometimes be reused with posterior samples. The optimization step may nevertheless be computationally difficult.\n\nReinforcement learning\n\nIn reinforcement learning, actions can change future states as well as immediate rewards. Posterior sampling can be extended by drawing an environment model, computing a policy for it, and following that policy for an episode or another specified interval.\n\nThis is often called posterior sampling for reinforcement learning. Keeping a sampled model fixed across multiple decisions can produce temporally coherent exploration. Independently resampling at every step can lead to different exploratory behavior."
    },
    {
      "heading": "Relationship to other methods",
      "text": "Thompson sampling differs from several common alternatives:\n\n- Greedy selection chooses the action with the highest current estimated reward and may perform no deliberate exploration.\n- Epsilon-greedy selection usually acts greedily but chooses a random action with a specified probability.\n- Upper-confidence-bound methods choose actions using optimistic estimates constructed from reward estimates and uncertainty bounds.\n- Bayes-optimal planning explicitly accounts for how present actions affect future knowledge and rewards.\n\nBoth Thompson sampling and upper-confidence-bound methods use uncertainty to guide exploration. Their mechanisms differ: Thompson sampling randomizes through posterior draws, while confidence-bound methods typically optimize an optimistic index."
    },
    {
      "heading": "Applications",
      "text": "Thompson sampling is used in online experimentation, advertising allocation, recommendation systems, and other settings where decisions and feedback arrive sequentially. It is especially natural when alternatives can be tested repeatedly and the objective is cumulative reward during learning.\n\nIn contrast to a conventional randomized controlled experiment with fixed allocation probabilities, Thompson sampling adapts allocation in response to outcomes. Reward optimization and statistical inference are distinct goals: adaptive allocation does not automatically support conventional fixed-design estimators, significance tests, or stopping rules without appropriate analysis.\n\nApplications may also involve delayed feedback, changing action sets, resource limits, or multiple objectives. Such features generally require modifications to the basic algorithm."
    },
    {
      "heading": "Limitations and practical considerations",
      "text": "Model and prior dependence. Posterior probabilities are conditional on the assumed model. A misspecified likelihood or an excessively concentrated, misleading prior can distort exploration. If the prior excludes the true environment, observations cannot simply recover it through ordinary posterior updating.\n\nApproximate inference. Exact posterior sampling is often unavailable. Methods such as Markov chain Monte Carlo, variational inference, and ensemble approximations can be used, but their uncertainty estimates affect exploration. An approximation that understates uncertainty may explore too little.\n\nNonstationarity. The basic formulation assumes fixed reward distributions. Under distribution shift, accumulating all past observations can make the model slow to respond. Discounting, sliding windows, or explicit models of temporal change alter this behavior, but also change the assumptions behind performance guarantees.\n\nDelayed and batched observations. Several actions may be selected before any new outcomes arrive. Decisions then rely on less current information, and sequential-update guarantees may not transfer unchanged.\n\nSafety and constraints. Randomized exploration can select inferior or unacceptable actions. Standard Thompson sampling does not itself impose safety requirements, fairness criteria, budget constraints, or limits on exposure to poor outcomes. These must be represented separately in the decision problem.\n\nReward specification. The method optimizes the supplied reward signal. High performance on that signal does not establish that it captures every relevant objective or long-term consequence."
    },
    {
      "heading": "History",
      "text": "William R. Thompson introduced the underlying idea in 1933 in work on choosing between two treatments when their success probabilities were uncertain. His proposal allocated choices according to the probability, given the available evidence, that one alternative was better than the other.\n\nThe method later became a central approach to bandit learning. Research in the early 2010s established strong empirical results and increasingly precise regret guarantees, helping extend its use to contextual decisions, structured action spaces, and reinforcement learning."
    }
  ],
  "links": [
    "randomized-algorithm",
    "probability-distribution",
    "bayesian-inference",
    "beta-distribution",
    "conjugate-prior",
    "artificial-neural-network",
    "mathematical-optimization",
    "reinforcement-learning",
    "recommender-system",
    "randomized-controlled-trial",
    "markov-chain-monte-carlo",
    "variational-inference",
    "distribution-shift"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}