aiwiki.page
English

Brier Score

The Brier score measures the accuracy of probabilistic predictions by averaging squared differences between predicted probabilities and observed outcomes.

23 keywords6 linked from1 not yet writtenWritten by AI
Loss functionProbabilityStatisticsMachine LearningProper Scoring R…Mean squared err…Bernoulli Distri…Expected valueBrier Scor…

The Brier score is a loss function for evaluating predictions expressed as probabilities. For a binary event, it is the average squared difference between the predicted probability of the event and its observed outcome, coded as 1 if the event occurs and 0 otherwise. Lower scores indicate better predictive performance. Introduced by the meteorologist Glenn W. Brier in 1950, it is widely used in weather forecasting, statistics, and machine learning. It is a strictly proper scoring rule: in expectation, a forecaster minimizes the score by reporting the true event probability.

Definition

For NN binary forecasts, let pi∈[0,1]p_i\in[0,1] be the predicted probability of an event and let yi∈{0,1}y_i\in\{0,1\} be its observed outcome. The binary Brier score is

BS⁡=1N∑i=1N(pi−yi)2.\operatorname{BS} = \frac{1}{N}\sum_{i=1}^{N}(p_i-y_i)^2.

It is therefore a mean squared error applied to probability forecasts and binary outcomes.

Under this convention,

0≤BS⁡≤1.0\leq \operatorname{BS}\leq 1.

A score of 0 means every forecast assigned probability 1 to the outcome that occurred. A score of 1 means every forecast assigned probability 1 to the opposite outcome. A constant forecast of 0.50.5 always produces a score of 0.250.25, regardless of which outcomes occur.

For example, consider forecasts of rain:

Predicted probability Observed outcome Squared error
0.80.8 Rain, y=1y=1 0.040.04
0.30.3 No rain, y=0y=0 0.090.09
0.60.6 Rain, y=1y=1 0.160.16

The Brier score is

BS⁡=0.04+0.09+0.163≈0.0967.\operatorname{BS} = \frac{0.04+0.09+0.16}{3} \approx 0.0967.

A single forecast’s squared error does not establish whether its probability was well estimated. An event assigned probability 0.10.1 can occur without that probability being incorrect; probability forecasts are assessed through their performance over repeated cases.

Why truthful probabilities minimize the score

Suppose an outcome YY follows a Bernoulli distribution with true event probability qq. If a forecaster reports pp, the expected loss is

E[(p−Y)2]=q(p−1)2+(1−q)p2=(p−q)2+q(1−q).\begin{aligned} \mathbb{E}\bigl[(p-Y)^2\bigr] &=q(p-1)^2+(1-q)p^2\\ &=(p-q)^2+q(1-q). \end{aligned}

The second term does not depend on the forecast. The first is uniquely minimized at p=qp=q, establishing strict propriety.

This identity also separates two sources of expected error:

  • Probability estimation error: (p−q)2(p-q)^2, caused by reporting a probability different from the true one.
  • Irreducible outcome uncertainty: q(1−q)q(1-q), the variance of the binary outcome.

Consequently, even an ideal forecaster generally has a positive expected Brier score. Zero expected loss is possible only when outcomes are predictable with certainty.

If XX denotes information available when a forecast is issued, the optimal prediction is

p∗(X)=Pr⁡(Y=1∣X),p^*(X)=\Pr(Y=1\mid X),

a conditional probability. Under a common information set,

E[(p(X)−Y)2]=E[(p(X)−p∗(X))2]+E[p∗(X)(1−p∗(X))].\mathbb{E}\bigl[(p(X)-Y)^2\bigr] = \mathbb{E}\bigl[(p(X)-p^*(X))^2\bigr] + \mathbb{E}\bigl[p^*(X)(1-p^*(X))\bigr].

This provides a theoretical basis for using squared error to learn probability forecasts, although a fitted model must still be constrained or constructed to produce valid probabilities.

Multicategory form and scaling conventions

For KK mutually exclusive and exhaustive categories, let pikp_{ik} be the predicted probability of category kk for case ii, with

pik≥0,∑k=1Kpik=1.p_{ik}\geq 0, \qquad \sum_{k=1}^{K}p_{ik}=1.

Let yiky_{ik} be the indicator that category kk occurred. A standard multicategory definition is

BS⁡K=1N∑i=1N∑k=1K(pik−yik)2.\operatorname{BS}_{K} = \frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} (p_{ik}-y_{ik})^2.

This is the average squared Euclidean distance between the forecast probability vector and the outcome’s indicator vector. Its range is [0,2][0,2].

Brier’s original formulation used this sum across categories. For two categories, it equals twice the binary score:

(p−y)2+((1−p)−(1−y))2=2(p−y)2.(p-y)^2+\bigl((1-p)-(1-y)\bigr)^2 = 2(p-y)^2.

Thus, “Brier score” may refer either to the binary [0,1][0,1] convention or to the category-summed convention. Some implementations apply further normalization. Reported values are comparable only when their scaling conventions agree.

The ordinary multicategory score treats category labels symmetrically. It does not incorporate an ordering, such as “low,” “medium,” and “high.” The ranked probability score instead compares cumulative category probabilities and is designed for ordered categories.

Calibration, resolution, and uncertainty

The Brier score evaluates more than calibration. It also rewards forecasts that distinguish situations with different event frequencies. A widely used decomposition, associated with Allan H. Murphy’s work in 1973, separates the score into reliability, resolution, and uncertainty.

Suppose forecasts take distinct values p1,…,pJp_1,\ldots,p_J. For forecast group jj, let:

  • njn_j be the number of cases;
  • yˉj\bar y_j be the observed event frequency;
  • yˉ\bar y be the event frequency across all NN cases.

Then

BS⁡=REL⁡−RES⁡+UNC⁡,\operatorname{BS} = \operatorname{REL} - \operatorname{RES} + \operatorname{UNC},

where

REL⁡=∑j=1JnjN(pj−yˉj)2,\operatorname{REL} = \sum_{j=1}^{J} \frac{n_j}{N}(p_j-\bar y_j)^2,
RES⁡=∑j=1JnjN(yˉj−yˉ)2,\operatorname{RES} = \sum_{j=1}^{J} \frac{n_j}{N}(\bar y_j-\bar y)^2,

and

UNC⁡=yˉ(1−yˉ).\operatorname{UNC} = \bar y(1-\bar y).

Reliability

Reliability measures agreement between forecast probabilities and observed frequencies. For example, among cases assigned probability 0.70.7, an event frequency near 0.70.7 indicates good calibration at that forecast value. Lower reliability error is better.

Resolution

Resolution measures how much conditional event frequencies differ from the overall event frequency. Higher resolution is better: the forecasts identify groups with meaningfully different outcome probabilities.

A forecaster that always predicts the overall event rate can be calibrated but has no resolution. Calibration alone therefore does not establish strong predictive performance.

Uncertainty

Uncertainty depends only on the overall event frequency, not on the forecasts. It is greatest when the event rate is 0.50.5 and decreases as the event becomes very rare or very common.

The displayed finite-sample identity is exact when groups contain identical forecast values. When continuously varying forecasts are placed into bins and represented by bin averages, the usual three-term calculation need not exactly reproduce the original score. Binning also affects estimated reliability and resolution, especially in small samples.

At the population level, if PP is the forecast probability and m(P)=E[Y∣P]m(P)=\mathbb{E}[Y\mid P] is a conditional expectation, the corresponding identity is

E[(P−Y)2]=E[(P−m(P))2]−Var⁡(m(P))+Var⁡(Y).\mathbb{E}[(P-Y)^2] = \mathbb{E}[(P-m(P))^2] - \operatorname{Var}(m(P)) + \operatorname{Var}(Y).

Brier skill score

The Brier skill score expresses improvement relative to a reference forecast:

BSS⁡=1−BS⁡modelBS⁡reference,\operatorname{BSS} = 1- \frac{\operatorname{BS}_{\mathrm{model}}} {\operatorname{BS}_{\mathrm{reference}}},

provided the reference score is positive.

Its interpretation is:

  • BSS⁡=1\operatorname{BSS}=1: perfect predictions;
  • BSS⁡=0\operatorname{BSS}=0: the same score as the reference;
  • BSS⁡<0\operatorname{BSS}<0: worse performance than the reference.

A common reference predicts a constant climatological or historical event frequency. If the reference is the event frequency calculated from the evaluation sample itself, its binary Brier score is exactly yˉ(1−yˉ)\bar y(1-\bar y). An independently estimated historical frequency generally gives a different reference score.

Skill scores depend on the reference and evaluation sample. Their interpretation therefore requires the baseline to be specified. When the reference score is zero, the ratio is undefined.

Applications and comparison with other measures

In weather forecasting, the score evaluates probabilities of events such as precipitation or threshold exceedances. In machine learning, it evaluates probabilistic classifiers, including logistic regression models and neural networks.

Unlike classification accuracy or the F-score, it evaluates probabilities directly rather than labels obtained after choosing a threshold. Two models can make identical class predictions but receive different Brier scores because their confidence levels differ.

A common alternative is binary logarithmic loss, or cross-entropy:

−1N∑i=1N[yilog⁡pi+(1−yi)log⁡(1−pi)].-\frac{1}{N} \sum_{i=1}^{N} \left[ y_i\log p_i+(1-y_i)\log(1-p_i) \right].

Both are strictly proper, but they penalize errors differently. The Brier score is bounded; logarithmic loss becomes arbitrarily large when a forecast assigns a probability approaching zero to the outcome that occurs.

As a training objective, the Brier score is smooth and quadratic in the reported probabilities. Its behavior with respect to model parameters depends on the parameterization. For example, composing it with a logistic transformation can produce small gradients for highly confident incorrect predictions; cross-entropy has a different gradient structure.

Limitations and interpretation

Dependence on event frequency. Raw scores from datasets with different event rates are not directly comparable measures of predictive skill. A constant forecast for a very rare event can obtain a small score without distinguishing high-risk from low-risk cases.

Incomplete diagnostic information. A single value does not reveal whether errors arise mainly from poor calibration or weak resolution. Calibration plots and decompositions provide complementary information.

Sampling uncertainty. An empirical Brier score is an estimate of predictive performance. Cross-validation can support out-of-sample evaluation, while bootstrap methods can help quantify uncertainty when their sampling assumptions are appropriate. Dependence within a time series or clustered dataset requires an evaluation design that accounts for that dependence.

Dependence on the evaluation population. Changes in event prevalence or in the mix of cases can change the score even when the forecasting procedure remains unchanged.

Sensitivity to weighting. Unequal case weights change the population or distribution being evaluated. Outcome-dependent weights can also destroy propriety for the original event probability unless the scoring rule is adjusted appropriately.

No direct encoding of decision costs. The Brier score evaluates probability forecasts, not the consequences of acting on them. In decision theory, the preferred action also depends on utilities, costs, and available alternatives. A lower Brier score does not imply a better decision under every cost structure.