{
  "id": "gibbs-sampling",
  "title": "Gibbs Sampling",
  "lang": "en",
  "source_lang": "en",
  "summary": "Gibbs sampling constructs a Markov chain by repeatedly drawing variables from their conditional distributions to approximate a joint distribution.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Gibbs sampler",
    "heat-bath sampling"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/gibbs-sampling/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/gibbs-sampling/",
      "json": "https://aiwiki.page/zh/gibbs-sampling.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Gibbs sampling is a Markov chain Monte Carlo method for generating dependent samples from a multivariate probability distribution. It updates one variable, or a group of variables, at a time by drawing from its distribution conditional on the current values of the remaining variables. Repetition produces a Markov chain whose stationary distribution is the desired joint distribution, provided that the conditional updates are correctly specified. Additional conditions determine whether the chain actually converges to that distribution from its starting point.\n\nThe method is especially important in Bayesian inference, where a posterior distribution may be difficult to sample directly even though its conditional distributions are simple. It also appears in statistical physics, image analysis, missing-data models, and machine learning. Its central advantage is the replacement of a difficult joint simulation problem by a sequence of smaller conditional simulation problems. Its principal limitation is that individually easy updates can still produce very slow exploration of the joint distribution."
    },
    {
      "heading": "Conditional distributions and the sampling algorithm",
      "text": "Let \\(X=(X_1,\\ldots,X_d)\\) have target joint density or mass function \\(\\pi(x_1,\\ldots,x_d)\\). Write \\(x_{-i}\\) for all coordinates except \\(x_i\\). The full conditional distribution of coordinate \\(i\\) is the conditional distribution\n\n\\[\\pi_i(x_i\\mid x_{-i}) = \\frac{\\pi(x_i,x_{-i})} {\\int \\pi(u,x_{-i})\\,du},\\]\n\nwith a sum replacing the integral for discrete variables. “Full” means that the conditioning includes every other coordinate, rather than an arbitrarily selected subset. In many models, factors not involving \\(x_i\\) cancel, making this distribution much simpler than the joint distribution.\n\nA systematic-scan Gibbs sampler begins at a state \\(x^{(0)}\\) within the target support and repeatedly updates coordinates in a fixed order. During sweep \\(t+1\\), it generates\n\n\\[X_i^{(t+1)} \\sim \\pi_i\\!\\left( \\,\\cdot\\mid X_1^{(t+1)},\\ldots,X_{i-1}^{(t+1)}, X_{i+1}^{(t)},\\ldots,X_d^{(t)} \\right), \\qquad i=1,\\ldots,d.\\]\n\nThus, each update uses the latest available values. A sweep comprises one update of every coordinate; an iteration may mean either one coordinate update or one complete sweep, so computational descriptions must distinguish the two. Random-scan Gibbs instead selects a coordinate at each step, usually with fixed positive selection probabilities. Random-permutation scanning chooses a fresh ordering for each sweep.\n\nThe target often need only be known up to a multiplicative constant. If \\(\\pi(x)=q(x)/Z\\), the global normalizing constant \\(Z\\) cancels from every full conditional. This is valuable when \\(Z\\) is a high-dimensional integral or a sum over exponentially many configurations. Nevertheless, each conditional must still be sampled correctly. Gibbs sampling does not automatically solve the problem of simulating a complicated one-dimensional distribution, nor does it make every conditional normalizing constant easy to evaluate.\n\nThe retained states are used as Monte Carlo samples. For a target quantity \\(\\mu=\\mathbb E_\\pi[h(X)]\\), a standard estimator is\n\n\\[\\widehat{\\mu}_N = \\frac{1}{N}\\sum_{t=1}^{N}h(X^{(t)}).\\]\n\nUnlike independent random sampling, neighboring Gibbs states generally remain correlated. Their dependence does not invalidate this estimator under appropriate ergodic conditions, but it changes its uncertainty and determines how much information a run contains.\n\nCoordinates need not be scalar. A block update draws a vector \\(X_B\\) jointly conditional on \\(X_{-B}\\). This extends the same principle without changing its foundation: the update must preserve the appropriate conditional distribution while leaving the complementary variables fixed."
    },
    {
      "heading": "Stationarity, reversibility, and convergence",
      "text": "The mathematical basis of Gibbs sampling is invariance under conditional resampling. Suppose \\(X\\) already has joint distribution \\(\\pi\\). If \\(X_i\\) is replaced by a new draw from \\(\\pi_i(\\cdot\\mid X_{-i})\\), the marginal distribution of \\(X_{-i}\\) is unchanged, and the new conditional distribution of \\(X_i\\) given \\(X_{-i}\\) is exactly the target conditional. The resulting vector therefore again has distribution \\(\\pi\\). In transition-kernel notation, every coordinate kernel \\(K_i\\) satisfies\n\n\\[\\pi K_i=\\pi.\\]\n\nConsequently, any fixed composition of these kernels also leaves \\(\\pi\\) invariant. This explains why a systematic sweep \\(K_1K_2\\cdots K_d\\) has the correct stationary distribution.\n\nEach individual Gibbs update also satisfies detailed balance. In density notation, for two states sharing the same complementary coordinates,\n\n\\[\\pi(x_{-i})\\pi_i(x_i\\mid x_{-i}) \\pi_i(y_i\\mid x_{-i})\\]\n\nis symmetric in \\(x_i\\) and \\(y_i\\). A random-scan kernel with fixed coordinate-selection probabilities is a mixture of reversible kernels and is therefore reversible. A systematic sweep, however, is generally not reversible: composing reversible updates does not necessarily preserve reversibility. Stationarity, not reversibility, is the essential requirement for correct long-run sampling.\n\nInvariance alone does not guarantee convergence from an arbitrary starting state. On a finite state space, irreducibility and aperiodicity provide standard sufficient conditions. Irreducibility means that every relevant state can be reached through allowed transitions; aperiodicity excludes deterministic cycling among classes of states. On general state spaces, corresponding convergence results use conditions such as positive Harris recurrence and aperiodicity. A strictly positive density on an appropriate product support is often helpful, but convergence must be considered in relation to the actual support and update scheme.\n\nA simple failure occurs when a two-variable binary distribution puts probability \\(1/2\\) on each of \\((0,0)\\) and \\((1,1)\\), with zero probability elsewhere. Given either coordinate, the other is determined. Single-coordinate Gibbs sampling cannot move between the two supported states, even though both belong to the target distribution. Updating both coordinates as one block removes this obstruction. Less extreme versions arise with narrow constraints, where transitions are possible but exceedingly rare.\n\nUnder suitable ergodicity and integrability conditions, a law of large numbers justifies averages along the chain. A central limit theorem requires additional assumptions and is not automatic. Some chains converge to their target yet have inadequate moment or mixing properties for conventional Monte Carlo standard errors. Thus, existence of a stationary distribution, convergence toward it, and reliable finite-run error assessment are distinct mathematical questions."
    },
    {
      "heading": "A Gaussian example and the geometry of slow mixing",
      "text": "A bivariate normal distribution gives an explicit illustration of Gibbs sampling and its limitations. Let \\(X\\) and \\(Y\\) each have mean zero and variance one, with correlation \\(\\rho\\), where \\(|\\rho|<1\\). Their joint distribution is a multivariate normal distribution, and the full conditionals are\n\n\\[X\\mid Y=y\\sim \\mathcal N(\\rho y,1-\\rho^2), \\qquad Y\\mid X=x\\sim \\mathcal N(\\rho x,1-\\rho^2).\\]\n\nA systematic sweep draws \\(X_{t+1}\\) given \\(Y_t\\), followed by \\(Y_{t+1}\\) given \\(X_{t+1}\\). Both draws are straightforward, regardless of how large \\(|\\rho|\\) is.\n\nWriting the two independent normal innovations explicitly gives\n\n\\[X_{t+1}=\\rho Y_t+\\sqrt{1-\\rho^2}\\,\\varepsilon_{t+1},\\]\n\n\\[Y_{t+1}=\\rho^2Y_t+ \\sqrt{1-\\rho^2}\\, \\bigl(\\rho\\varepsilon_{t+1}+\\eta_{t+1}\\bigr),\\]\n\nwhere \\(\\varepsilon_{t+1}\\) and \\(\\eta_{t+1}\\) are independent standard normal variables. The sequence \\(Y_t\\), recorded after complete sweeps, is therefore an autoregressive process with coefficient \\(\\rho^2\\). At stationarity, its lag-\\(k\\) autocorrelation is \\(\\rho^{2k}\\). This exact calculation shows why easily sampled conditionals do not imply efficient joint exploration.\n\nFor the sample mean of \\(Y\\), the integrated autocorrelation time is\n\n\\[\\tau = 1+2\\sum_{k=1}^{\\infty}\\rho^{2k} = \\frac{1+\\rho^2}{1-\\rho^2}.\\]\n\nWhen \\(\\rho=0.99\\), this quantity is approximately \\(99.5\\). Thus, for this particular estimand, about 10,000 stationary sweeps contain roughly the same asymptotic information as 100 independent draws. This is not a universal conversion for the entire chain: nonlinear functions and other coordinates can have different autocorrelation behavior.\n\nGeometrically, strong correlation concentrates probability in a long, narrow ellipse. Single-coordinate moves travel parallel to the coordinate axes. Because the ellipse is narrow across those directions, successive updates tend to make only small progress along its long axis. A joint block draw can cross the ellipse directly. A change of coordinates that approximately aligns the axes with the ellipse can also improve exploration. In larger Gaussian models, the same phenomenon is expressed through the covariance or precision matrix and the iteration matrices associated with coordinate updates.\n\nThe limiting case clarifies the role of support. At \\(|\\rho|=1\\), the distribution becomes singular and lies on a line. The conditional variances vanish, so ordinary scalar Gibbs updates become deterministic and cannot explore that line. Although this limit differs from every nonsingular case, it explains the progressively slower behavior as the conditional variances approach zero."
    },
    {
      "heading": "Bayesian computation with conjugate models",
      "text": "Gibbs sampling is especially convenient when a Bayesian model has recognizable full conditional distributions. Consider observations \\(y_1,\\ldots,y_n\\) satisfying\n\n\\[y_j\\mid\\mu,\\tau\\sim \\mathcal N(\\mu,\\tau^{-1}),\\]\n\nwhere \\(\\tau\\) is the precision. Take independent priors\n\n\\[\\mu\\sim\\mathcal N(m_0,s_0^2), \\qquad \\tau\\sim\\operatorname{Gamma}(a_0,b_0),\\]\n\nwith the gamma distribution parameterized by shape and rate, so its density is proportional to \\(\\tau^{a_0-1}e^{-b_0\\tau}\\). Specifying the parameterization matters because software also commonly uses a shape-and-scale convention.\n\nMultiplying the likelihood by the prior distributions and collecting terms yields\n\n\\[\\mu\\mid\\tau,y\\sim\\mathcal N(m_\\mu,V_\\mu),\\]\n\nwhere\n\n\\[V_\\mu=(n\\tau+s_0^{-2})^{-1}, \\qquad m_\\mu=V_\\mu\\left(\\tau\\sum_{j=1}^{n}y_j+s_0^{-2}m_0\\right).\\]\n\nThe precision update is\n\n\\[\\tau\\mid\\mu,y \\sim \\operatorname{Gamma} \\left( a_0+\\frac n2,\\, b_0+\\frac12\\sum_{j=1}^{n}(y_j-\\mu)^2 \\right).\\]\n\nAlternating these normal and gamma draws produces a Gibbs sampler for the joint posterior. Although the chosen priors are independent rather than a single normal-gamma joint prior, they give standard full conditionals and are often described as conditionally conjugate.\n\nThe resulting simulation provides more than point estimates. Averages estimate posterior expectations, empirical quantiles estimate credible intervals, and simulated future observations approximate the posterior predictive distribution. For example, after drawing \\((\\mu^{(t)},\\tau^{(t)})\\), a future observation can be generated from \\(\\mathcal N(\\mu^{(t)},(\\tau^{(t)})^{-1})\\). This propagates uncertainty in both parameters rather than substituting fixed estimates.\n\nThe same construction scales to hierarchical models. In a model with group-specific means, observation noise, and a population distribution for those means, separate updates can target each group effect, the population mean, and variance components. Conditional independence often permits all group effects to be sampled simultaneously once the hyperparameters are fixed. Hierarchical linear regression similarly produces multivariate normal coefficient updates under normal priors, with gamma or inverse-gamma updates for suitable variance parameters.\n\nConjugacy is a computational convenience, not a requirement of Bayesian inference or a guarantee of good mixing. Strong dependence between group effects and variance parameters can make a fully conjugate sampler inefficient. Moreover, apparently standard full conditional distributions do not by themselves establish that the joint posterior is proper when improper priors are used. The target must define a finite probability distribution before posterior simulation has its usual interpretation."
    },
    {
      "heading": "Blocking, collapsing, and auxiliary variables",
      "text": "Blocking replaces several scalar updates with one joint conditional update. Its purpose is to allow strongly dependent variables to move together. In Gaussian regression, for example, sampling the entire coefficient vector from its multivariate normal full conditional can be far more effective than updating individual coefficients. The cost is computational: a dense covariance factorization may require substantial time and memory. Cholesky decomposition, sparse precision matrices, and reusable factorizations determine whether the larger move is worthwhile. Efficiency depends on information gained per unit of computation, not simply on the number of iterations.\n\nCollapsing removes selected variables analytically before simulation. If the target is \\(\\pi(x,z)\\) and\n\n\\[\\pi(x)=\\int \\pi(x,z)\\,dz\\]\n\ncan be evaluated sufficiently well, a sampler may operate only on \\(x\\). The eliminated variables no longer need to be explored by the chain, and their removal can reduce dependence. They can later be reconstructed through draws from \\(\\pi(z\\mid x)\\). Collapsing is particularly useful in models with conjugate priors, where the necessary integrals have closed forms. It can, however, turn local conditional updates into globally coupled ones, increasing each update’s cost.\n\nA related variance-reduction idea follows the Rao–Blackwell theorem. Suppose the desired expectation is \\(\\mathbb E[h(X,Z)]\\), and the conditional expectation \\(g(X)=\\mathbb E[h(X,Z)\\mid X]\\) is available. Averaging \\(g(X^{(t)})\\) analytically integrates out some simulation noise. The ordinary law of total variance explains the reduction in variance for an individual stationary draw. For correlated Monte Carlo averages, however, asymptotic efficiency also depends on the serial dependence introduced by the chosen sampler; a simple marginal-variance comparison does not settle every case.\n\nData augmentation takes the opposite-looking approach: it introduces auxiliary variables to simplify conditional distributions. If an expanded target \\(\\widetilde\\pi(x,z)\\) has marginal \\(\\pi(x)\\), alternating updates of \\(z\\mid x\\) and \\(x\\mid z\\) yields a Gibbs sampler whose \\(x\\)-states target the desired distribution. In binary probit regression, latent Gaussian responses can be introduced so that the observed binary outcomes record their signs. The latent responses then have truncated normal conditionals, while regression coefficients have a normal conditional under a normal prior.\n\nThese strategies interact with parameterization. Centered hierarchical parameters may be highly dependent on population-level parameters when data are weak, whereas noncentered representations express group effects through standardized auxiliary variables. Neither parameterization is uniformly best. Partially collapsed samplers require particular care: removing conditioning variables from only some updates can invalidate the target unless the ordering and retained variables are coordinated. A collection of individually plausible conditional steps is not sufficient evidence that the full algorithm is correct."
    },
    {
      "heading": "Hybrid updates, scan order, and parallel computation",
      "text": "Pure Gibbs sampling assumes that every chosen conditional distribution can be sampled exactly. When this is impractical, a common alternative is to embed a Metropolis–Hastings transition within the coordinate update. Given \\(x_{-i}\\), a proposal \\(y_i\\sim q_i(\\cdot\\mid x_i,x_{-i})\\) is accepted with probability\n\n\\[\\alpha = \\min\\left\\{ 1,\\, \\frac{ \\pi(y_i,x_{-i})q_i(x_i\\mid y_i,x_{-i}) }{ \\pi(x_i,x_{-i})q_i(y_i\\mid x_i,x_{-i}) } \\right\\}.\\]\n\nThe conditional normalizing constant cancels. If each embedded transition preserves its conditional target, composing these transitions gives a valid Metropolis-within-Gibbs algorithm. One need not run the inner transition to equilibrium before moving to another coordinate.\n\nAn exact Gibbs draw can itself be viewed as a Metropolis–Hastings proposal drawn from the full conditional; its acceptance probability is one. This relationship does not mean Gibbs sampling is automatically more efficient than a proposal-based method. A carefully chosen joint proposal can cross strongly correlated regions more effectively than exact but narrowly constrained coordinate draws. Hybrid algorithms may therefore combine Gibbs updates for discrete variables or conjugate blocks with Hamiltonian Monte Carlo or other transitions for continuous blocks.\n\nScan order also affects behavior. Fixed systematic scans, random scans, and random permutations all preserve the target under their standard definitions, but they can have different convergence rates and computational costs. State-dependent selection probabilities require special analysis: selecting a coordinate according to the current state and then applying its ordinary Gibbs update does not generally preserve \\(\\pi\\). Likewise, adaptation based on the history of the chain is not justified merely because every fixed choice of tuning parameters would be valid.\n\nParallel updates are exact when the relevant conditional factorization supports them. If a set of variables is mutually conditionally independent given the complement, their joint conditional distribution is a product, so they can be drawn simultaneously. In a graphical model, a proper coloring of the dependency graph partitions variables into such groups. Each color class can be updated in parallel, followed by the next class. A checkerboard update on a nearest-neighbor square lattice is a familiar example.\n\nUpdating dependent variables simultaneously using outdated neighbor values is a different algorithm. It generally does not have the same invariant distribution as sequential Gibbs sampling. Asynchronous implementations may be useful under specific conditions, but their validity or approximation error requires separate arguments. Numerical details matter as well: inaccurate conditional samplers, underflow in categorical probabilities, inconsistent parameter conventions, or errors in cached sufficient statistics can change the implemented transition kernel even when the mathematical specification is correct."
    },
    {
      "heading": "Monte Carlo error and convergence assessment",
      "text": "A Gibbs run has two conceptually different sources of error: initial-distribution effects and finite-simulation variability. Starting away from equilibrium can distort early estimates, motivating a warm-up or burn-in period. Discarding an arbitrary number of initial states does not establish convergence, however. A chain can remain trapped in one region long after its trace appears locally stable. Conversely, no burn-in is needed to remove initialization bias if the starting state is already exactly distributed according to the target.\n\nFor a stationary scalar sequence \\(H_t=h(X^{(t)})\\), suppose the autocovariances \\(\\gamma_k=\\operatorname{Cov}(H_t,H_{t+k})\\) are sufficiently summable. The asymptotic variance of its sample average is governed by\n\n\\[\\sigma_{\\mathrm{MC}}^2 = \\gamma_0+2\\sum_{k=1}^{\\infty}\\gamma_k, \\qquad \\operatorname{Var}(\\widehat\\mu_N) \\approx \\frac{\\sigma_{\\mathrm{MC}}^2}{N}.\\]\n\nThe effective sample size for estimating this expectation is commonly expressed as\n\n\\[N_{\\mathrm{eff}} = N\\frac{\\gamma_0}{\\sigma_{\\mathrm{MC}}^2}.\\]\n\nIt compares the chain’s precision with that of independent draws having the same marginal variance. It is a property of an estimand and a run, not a single intrinsic sample count applicable to every posterior quantity.\n\nMonte Carlo standard errors can be estimated through batch means, spectral methods, or suitable autocorrelation estimators. These quantify simulation error, which is different from posterior uncertainty. A wide posterior interval can be estimated with a small Monte Carlo error, while a narrow posterior interval may still be estimated poorly if the chain mixes slowly. Rare-event probabilities and tail quantiles often need substantially more computation than central expectations.\n\nMultiple chains initialized in different regions provide information that a single trajectory cannot. Between-chain and within-chain comparisons, including rank-normalized and split forms of the potential scale reduction statistic, are widely used to detect failures of mixing. Trace plots can reveal drift, sticking, or different chains occupying different modes. None of these checks proves that all important regions have been visited. In a multimodal target, several chains can agree simply because they all missed the same mode.\n\nRoutine thinning—retaining, for example, only every tenth state—usually sacrifices statistical information when computation rather than storage is the limiting resource. It can be reasonable when storing or processing each state is unusually expensive, but it does not fix poor exploration. More direct remedies include changing parameterization, improving blocks, modifying the transition scheme, or increasing computation. The relevant comparison is ultimately uncertainty in the quantities of interest per unit of time and memory."
    },
    {
      "heading": "Statistical physics, graphical models, and latent-variable applications",
      "text": "In statistical physics, Gibbs sampling is often called heat-bath sampling. Consider an Ising model with spins \\(s_i\\in\\{-1,+1\\}\\), interaction energy\n\n\\[E(s)=-\\sum_{(i,j)}J_{ij}s_is_j-\\sum_i h_is_i,\\]\n\nand equilibrium probability proportional to \\(\\exp[-\\beta E(s)]\\), where \\(\\beta\\) is inverse temperature. Holding all other spins fixed, the probability of setting spin \\(i\\) to \\(+1\\) is\n\n\\[\\Pr(s_i=+1\\mid s_{-i}) = \\frac{1}{ 1+\\exp\\left[-2\\beta\\left(h_i+\\sum_jJ_{ij}s_j\\right)\\right] }.\\]\n\nOnly neighboring interactions enter this update. At low temperatures or near critical regimes, local updates can explore the system slowly because large-scale configurations change through many coordinated local events.\n\nThis locality generalizes to a Markov random field and other probabilistic graphical models. For a factorization\n\n\\[\\pi(x)\\propto\\prod_a\\psi_a(x_a),\\]\n\nthe full conditional of \\(x_i\\) contains only factors involving that variable:\n\n\\[\\pi(x_i\\mid x_{-i}) \\propto \\prod_{a:\\,i\\in a}\\psi_a(x_a).\\]\n\nThis cancellation makes Gibbs updates computationally attractive in sparse models. In a Bayesian network, the full conditional depends on a node’s parents, children, and the other parents of its children. Graphical conditional independence therefore determines both the local computation and the possibilities for blocking or parallel execution.\n\nLatent-variable models provide another major application. In a finite Gaussian mixture model, the sampler can alternate between assigning each observation to a component and updating component parameters and mixture weights. Given assignments, many parameter updates become standard conjugate calculations. However, symmetric component priors imply that permuting component labels leaves the posterior unchanged. This label-switching symmetry complicates interpretation of component-specific averages, while well-separated clustering alternatives can create additional barriers to exploration.\n\nIn topic modeling, latent Dirichlet allocation uses latent topic assignments together with document-specific topic proportions and topic-specific word probabilities. With Dirichlet priors, the proportions and word probabilities can be integrated out, producing a collapsed Gibbs sampler for topic assignments. For symmetric hyperparameters \\(\\alpha\\) and \\(\\eta\\), a token \\(i\\) in document \\(d\\), with word type \\(v\\), has conditional topic probabilities proportional to\n\n\\[\\Pr(z_i=k\\mid z_{-i},w) \\propto (n_{dk}^{-i}+\\alpha) \\frac{n_{kv}^{-i}+\\eta}{n_k^{-i}+V\\eta},\\]\n\nwhere the counts exclude the token being updated and \\(V\\) is the vocabulary size.\n\nMissing-data problems use a similar alternating construction: draw missing values conditional on parameters and observed data, then draw parameters conditional on the completed data. This connects Gibbs sampling to missing-data imputation and uncertainty propagation. Unlike the expectation–maximization algorithm, which uses conditional expectations within an optimization procedure, Gibbs sampling generates random conditional draws to explore a distribution over unknown quantities."
    },
    {
      "heading": "Historical development and methodological boundaries",
      "text": "Gibbs sampling takes its name from the Gibbs distributions of statistical mechanics, associated with J. Willard Gibbs, rather than from an algorithm devised by Gibbs himself. Its computational ancestry lies in simulation methods for interacting physical systems, where local changes to a configuration can be calculated without evaluating the full partition function. Heat-bath updates became a natural counterpart to acceptance–rejection transition rules in the Monte Carlo study of lattice models.\n\nA landmark statistical presentation was Stuart Geman and Donald Geman’s 1984 work on stochastic relaxation, Gibbs distributions, and Bayesian image restoration. Their construction linked local conditional simulation, spatial image models, and simulated annealing. In image restoration, pixels or associated latent variables interact through a spatial prior, while observed noisy intensities contribute likelihood terms. Sampling at a fixed temperature targets a distribution; gradually lowering temperature instead serves an optimization objective. These are related uses of local stochastic updates, but their goals and convergence questions differ.\n\nThe method’s adoption in mainstream Bayesian statistics accelerated around 1990. Alan Gelfand and Adrian Smith’s 1990 paper on sampling-based approaches to marginal densities demonstrated how conditional simulation could replace difficult analytical integration in a broad range of statistical models. Martin Tanner and Wing Hung Wong’s 1987 work on data augmentation also helped establish simulation-based methods for posterior computation. The subsequent spread of hierarchical modeling and general-purpose Bayesian software made conditional simulation a routine computational tool.\n\nThe boundary between Gibbs sampling and other iterative methods is important. Coordinate optimization replaces each variable by a value that optimizes an objective conditional on the rest; Gibbs sampling draws from a conditional distribution and preserves uncertainty. Mean-field variational inference updates an approximating distribution under a factorization assumption rather than following the same Markov transition. Similar-looking alternating calculations can therefore have fundamentally different mathematical targets.\n\nFinally, full conditionals cannot always be specified independently. They must be compatible with a common joint distribution. For example, if one proposes Gaussian conditionals\n\n\\[X\\mid Y=y\\sim\\mathcal N(ay,s_x^2), \\qquad Y\\mid X=x\\sim\\mathcal N(bx,s_y^2),\\]\n\ncompatibility with a nonsingular joint Gaussian requires, among other conditions,\n\n\\[\\frac{a}{s_x^2}=\\frac{b}{s_y^2}, \\qquad ab<1.\\]\n\nThese restrictions reflect symmetry and positive definiteness of the joint precision matrix. Cycling through incompatible conditionals still defines a computational procedure, but its behavior need not correspond to the proposed probabilistic model and may depend on update order. Related issues arise in conditional models for imputation, where convenient regression specifications must be distinguished from full conditionals derived from a coherent joint target."
    }
  ],
  "links": [
    "markov-chain-monte-carlo",
    "probability-distribution",
    "markov-chain",
    "bayesian-inference",
    "posterior-distribution",
    "conditional-probability",
    "integral",
    "monte-carlo-method",
    "stationary-distribution",
    "detailed-balance",
    "law-of-large-numbers",
    "central-limit-theorem",
    "correlation",
    "multivariate-normal-distribution",
    "autoregressive-model",
    "autocorrelation",
    "likelihood-function",
    "prior-distribution",
    "credible-interval",
    "posterior-predictive-distribution",
    "linear-regression",
    "cholesky-decomposition",
    "conjugate-prior",
    "rao-blackwell-theorem",
    "data-augmentation",
    "metropolis-hastings-algorithm",
    "hamiltonian-monte-carlo",
    "effective-sample-size",
    "standard-error",
    "ising-model",
    "markov-random-field",
    "probabilistic-graphical-model",
    "bayesian-network",
    "gaussian-mixture-model",
    "dirichlet-distribution",
    "missing-data-imputation",
    "expectation-maximization-algorithm",
    "simulated-annealing",
    "variational-inference"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}