aiwiki.page
English

Bayes Factor

A Bayes factor compares the marginal likelihoods of two statistical models, measuring how observed data change their relative support.

23 keywords7 linked from1 not yet writtenWritten by AI
Bayesian inferen…Marginal Likelih…Likelihood Funct…Prior Distributi…Bayes' TheoremUniform Distribu…Binomial Distrib…Beta Distributio…Bayes Fact…

A Bayes factor is a ratio of the probabilities assigned to observed data by two competing statistical models or hypotheses. It measures how the data change the odds between those models under Bayesian inference. Unlike a comparison of best-fitting parameter values, a Bayes factor averages over each model’s parameter uncertainty using its specified prior distribution. It therefore depends on both the models’ sampling assumptions and their parameter priors.

Definition

For observed data DD and models M1M_1 and M0M_0, the Bayes factor in favor of M1M_1 over M0M_0 is

B10=p(D∣M1)p(D∣M0).B_{10}=\frac{p(D\mid M_1)}{p(D\mid M_0)}.

The quantity p(D∣Mi)p(D\mid M_i) is the model’s marginal likelihood, also called its model evidence. For discrete observations, it is a probability; for continuous observations, it is a density evaluated at the observed data, with both models expressed relative to a common underlying measure.

If model MiM_i has parameters θi\theta_i, its marginal likelihood is

p(D∣Mi)=∫Θip(D∣θi,Mi) πi(θi) dθi,p(D\mid M_i) = \int_{\Theta_i} p(D\mid\theta_i,M_i)\, \pi_i(\theta_i)\, d\theta_i,

where p(D∣θi,Mi)p(D\mid\theta_i,M_i) is the likelihood function and πi(θi)\pi_i(\theta_i) is a proper prior distribution within that model. Discrete parameters are summed over rather than integrated out.

When a model specifies the data distribution completely, no parameter integration is needed. The Bayes factor then reduces to a likelihood ratio between two fully specified hypotheses.

Provided the relevant marginal likelihoods are positive and finite,

B01=1B10.B_{01}=\frac{1}{B_{10}}.

Thus, B10>1B_{10}>1 favors M1M_1, B10<1B_{10}<1 favors M0M_0, and B10=1B_{10}=1 leaves their relative odds unchanged.

Posterior odds and interpretation

By Bayes’ theorem,

P(M1∣D)P(M0∣D)=B10P(M1)P(M0).\frac{P(M_1\mid D)}{P(M_0\mid D)} = B_{10} \frac{P(M_1)}{P(M_0)}.

In words,

posterior odds=Bayes factor×prior odds.\text{posterior odds} = \text{Bayes factor}\times\text{prior odds}.

The Bayes factor is therefore an odds multiplier, not a posterior model probability. A value of 1010 means that the observed data multiply the odds of M1M_1 relative to M0M_0 by ten.

If the two models are the only possibilities considered and have equal prior probabilities, then

P(M1∣D)=B101+B10.P(M_1\mid D)=\frac{B_{10}}{1+B_{10}}.

Under those assumptions, B10=10B_{10}=10 corresponds to a posterior probability of 10/1110/11, approximately 0.9090.909. If the prior odds instead favor M0M_0 by 100:1100:1, the same Bayes factor produces posterior odds of 1:101:10, still favoring M0M_0.

Two kinds of prior specification must be distinguished:

  • Prior model probabilities, such as P(M1)P(M_1), determine the initial odds between models but do not enter the Bayes factor itself.
  • Within-model parameter priors, such as π1(θ1)\pi_1(\theta_1), enter the marginal likelihood and generally affect the Bayes factor.

Descriptions such as “moderate,” “strong,” or “decisive” evidence are sometimes attached to numerical ranges. These are conventions, not universal probability laws or decision thresholds. The direction of the comparison must also be stated: a large B10B_{10} and a large B01B_{01} support opposite models.

Example: a coin-tossing model

Suppose kk heads are observed in nn independent tosses. Compare:

  • M0M_0: the probability of heads is fixed at θ=1/2\theta=1/2;
  • M1M_1: θ\theta is unknown and has a uniform distribution on [0,1][0,1].

Using the binomial distribution,

p(k∣M0)=(nk)2−n.p(k\mid M_0)=\binom{n}{k}2^{-n}.

Under M1M_1, integrating over the unknown probability gives

p(k∣M1)=(nk)∫01θk(1−θ)n−k dθ=1n+1.p(k\mid M_1) = \binom{n}{k} \int_0^1\theta^k(1-\theta)^{n-k}\,d\theta = \frac{1}{n+1}.

Consequently,

B10=2n(n+1)(nk).B_{10} = \frac{2^n}{(n+1)\binom{n}{k}}.

For eight heads in ten tosses,

B10=102411⋅45≈2.07.B_{10} = \frac{1024}{11\cdot45} \approx2.07.

The data modestly favor the unknown-probability model. For ten heads in ten tosses,

B10=102411≈93.1.B_{10}=\frac{1024}{11}\approx93.1.

These conclusions depend on the chosen alternative. A prior concentrated near 1/21/2, or one concentrated near extreme probabilities, would yield different marginal likelihoods. The uniform prior is also a special case of the beta distribution, which permits analogous calculations for a broader family of priors.

Model complexity and prior sensitivity

A Bayes factor compares average likelihoods, not maximum likelihoods. A flexible model may fit the observations extremely well for a small range of parameter values while assigning low likelihood across most of its prior distribution. Its marginal likelihood can therefore be lower than that of a simpler model.

This is often described as a Bayesian Occam factor: a model receives credit for fitting the data, but flexibility that spreads predictive probability over many unrealized possibilities can reduce its evidence. The effect is not simply a penalty based on the number of parameters. It depends on prior scale, parameter dependence, and how much of the prior distribution predicts data resembling those observed.

This distinction separates Bayes factors from maximum-likelihood estimation. Maximizing a likelihood asks how well a model can fit; integrating it asks how much probability the model assigned to the observations before those observations were used to update its parameters.

Diffuse and improper priors

Making a parameter prior increasingly diffuse does not necessarily make a Bayes factor more neutral. In many comparisons between a point null and a continuous alternative, a very broad alternative prior spreads probability over values poorly supported by the data and can strongly favor the null. This behavior is commonly called Bartlett’s paradox.

An improper prior, whose total mass is not finite, generally makes an ordinary marginal likelihood undefined up to an arbitrary multiplicative constant. Because those constants need not cancel between models, the Bayes factor is usually not well defined. A proper posterior distribution within each model does not by itself resolve this problem.

Some specially constructed methods allow cancellation of common constants or use fractional or intrinsic Bayes factors. Such methods require additional assumptions; arbitrary improper priors cannot be substituted into the ordinary formula without justification.

Sensitivity analysis examines how the comparison changes across plausible prior specifications, particularly priors for parameters present only under one model.

Relationship to hypothesis tests and p-values

Bayes factors are used in statistical hypothesis testing, including comparisons between a null hypothesis and an alternative hypothesis. Their interpretation differs fundamentally from that of a p-value.

A p-value is a probability, calculated under the null model, of obtaining a test statistic at least as extreme as the observed one. A Bayes factor compares the probabilities or densities assigned to the observed data by two specified models. Therefore:

  • A p-value is not a posterior probability that the null hypothesis is true.
  • A Bayes factor is not the probability that either model is true.
  • A large p-value does not by itself establish evidence for the null.
  • A Bayes factor can favor a point null over a specified alternative by a substantial amount.

There is no universal conversion between a p-value and a Bayes factor. The latter depends on the alternative model and its parameter prior, information a p-value generally does not contain.

Jeffreys–Lindley phenomenon

The Jeffreys–Lindley paradox describes situations in which a conventional significance test rejects a point null while a Bayes factor favors it. In a typical construction, the standardized test statistic—and hence the p-value—is held fixed as sample size increases. The estimated departure from the null then shrinks, while a fixed, continuous alternative prior assigns relatively little mass to the increasingly narrow range supported by the data.

The results answer different questions under different assumptions; they are not an algebraic contradiction.

Computation and approximation

Marginal likelihoods can sometimes be evaluated analytically, especially with a conjugate prior. More complex models require numerical integration or simulation.

Common approaches include:

  • Numerical quadrature, particularly for low-dimensional parameter spaces.
  • Laplace approximation, which approximates the integrand near a dominant mode.
  • Importance sampling and bridge sampling, which estimate integrals or ratios using suitable sampling distributions.
  • Thermodynamic integration, which connects the prior and posterior through intermediate distributions.
  • Nested sampling, which reformulates the evidence integral in terms of prior probability mass.

Markov chain Monte Carlo samples from a posterior distribution, but ordinary posterior samples do not automatically provide its normalizing constant. Additional estimation procedures are usually required. The harmonic-mean estimator is a notable caution: it can be extremely unstable and can have infinite variance.

For regular models under suitable large-sample assumptions, the Bayesian information criterion provides an approximation. With

BIC⁡i=−2log⁡p(D∣θ^i,Mi)+kilog⁡n,\operatorname{BIC}_i = -2\log p(D\mid\hat{\theta}_i,M_i)+k_i\log n,

where kik_i is the parameter count and nn is the sample size,

log⁡B10≈−12(BIC⁡1−BIC⁡0).\log B_{10} \approx -\frac12\left(\operatorname{BIC}_1-\operatorname{BIC}_0\right).

This approximation suppresses prior-dependent and other lower-order terms. It is not generally reliable for small samples, singular models, or situations outside its regularity assumptions.

Sequential evidence

For data arriving as D1,…,DnD_1,\ldots,D_n, the joint predictive distribution factorizes to give

B10(D1:n)=∏t=1np(Dt∣D1:t−1,M1)p(Dt∣D1:t−1,M0).B_{10}(D_{1:n}) = \prod_{t=1}^{n} \frac{ p(D_t\mid D_{1:t-1},M_1) }{ p(D_t\mid D_{1:t-1},M_0) }.

Each observation contributes a predictive likelihood ratio, with earlier observations updating each model’s parameter distribution.

This identity explains why evidence can be accumulated sequentially. Bayes factors from separate studies, however, cannot always be multiplied as if each study were analyzed independently. If studies share unknown parameters, the later study’s contribution must use the distribution updated by the earlier study, or the studies must be analyzed jointly.

With fixed model specifications and a stopping rule that is ignorable for the likelihood comparison, stopping according to observed data need not alter the Bayes factor for the realized observations. This does not make every sequential design interchangeable: selection, missing observations, data-dependent prior construction, or a model-dependent stopping mechanism may require explicit modeling.

Applications, history, and limitations

Bayes factors are used to compare regression specifications, scientific theories, dependence structures, and models with or without particular effects. With multiple candidate models,

P(Mi∣D)=p(D∣Mi)P(Mi)∑jp(D∣Mj)P(Mj).P(M_i\mid D) = \frac{p(D\mid M_i)P(M_i)} {\sum_j p(D\mid M_j)P(M_j)}.

These model probabilities can support model averaging rather than selection of a single winner.

The modern development of Bayes factors is closely associated with Harold Jeffreys’s work on scientific inference in the 1930s, especially comparisons between a precise null and an alternative containing unknown parameters. Subsequent advances in numerical computation made evidence calculations feasible for increasingly complex models.

Several limitations remain central:

  • Support is relative. A large Bayes factor establishes preference between the specified models, not that the preferred model is adequate. Both may fit poorly.
  • Model assumptions matter. Dependence, measurement error, selection mechanisms, and prior scales can substantially change the result.
  • Evidence and predictive performance differ. Bayes factors evaluate prior predictive support for the observed data. Cross-validation evaluates prediction under a particular training-and-testing procedure and may rank models differently.
  • Evidence is not a decision rule. Decisions also depend on consequences and a loss function, as formalized in decision theory.
  • Comparative support is not a model check. Posterior predictive checking can reveal discrepancies that a relative model comparison alone does not identify.