The Brier score is a loss function for evaluating predictions expressed as probabilities. For a binary event, it is the average squared difference between the predicted probability of the event and its observed outcome, coded as 1 if the event occurs and 0 otherwise. Lower scores indicate better predictive performance. Introduced by the meteorologist Glenn W. Brier in 1950, it is widely used in weather forecasting, statistics, and machine learning. It is a strictly proper scoring rule: in expectation, a forecaster minimizes the score by reporting the true event probability.
Definition
For binary forecasts, let be the predicted probability of an event and let be its observed outcome. The binary Brier score is
It is therefore a mean squared error applied to probability forecasts and binary outcomes.
Under this convention,
A score of 0 means every forecast assigned probability 1 to the outcome that occurred. A score of 1 means every forecast assigned probability 1 to the opposite outcome. A constant forecast of always produces a score of , regardless of which outcomes occur.
For example, consider forecasts of rain:
| Predicted probability | Observed outcome | Squared error |
|---|---|---|
| Rain, | ||
| No rain, | ||
| Rain, |
The Brier score is
A single forecast’s squared error does not establish whether its probability was well estimated. An event assigned probability can occur without that probability being incorrect; probability forecasts are assessed through their performance over repeated cases.
Why truthful probabilities minimize the score
Suppose an outcome follows a Bernoulli distribution with true event probability . If a forecaster reports , the expected loss is
The second term does not depend on the forecast. The first is uniquely minimized at , establishing strict propriety.
This identity also separates two sources of expected error:
- Probability estimation error: , caused by reporting a probability different from the true one.
- Irreducible outcome uncertainty: , the variance of the binary outcome.
Consequently, even an ideal forecaster generally has a positive expected Brier score. Zero expected loss is possible only when outcomes are predictable with certainty.
If denotes information available when a forecast is issued, the optimal prediction is
a conditional probability. Under a common information set,
This provides a theoretical basis for using squared error to learn probability forecasts, although a fitted model must still be constrained or constructed to produce valid probabilities.
Multicategory form and scaling conventions
For mutually exclusive and exhaustive categories, let be the predicted probability of category for case , with
Let be the indicator that category occurred. A standard multicategory definition is
This is the average squared Euclidean distance between the forecast probability vector and the outcome’s indicator vector. Its range is .
Brier’s original formulation used this sum across categories. For two categories, it equals twice the binary score:
Thus, “Brier score” may refer either to the binary convention or to the category-summed convention. Some implementations apply further normalization. Reported values are comparable only when their scaling conventions agree.
The ordinary multicategory score treats category labels symmetrically. It does not incorporate an ordering, such as “low,” “medium,” and “high.” The ranked probability score instead compares cumulative category probabilities and is designed for ordered categories.
Calibration, resolution, and uncertainty
The Brier score evaluates more than calibration. It also rewards forecasts that distinguish situations with different event frequencies. A widely used decomposition, associated with Allan H. Murphy’s work in 1973, separates the score into reliability, resolution, and uncertainty.
Suppose forecasts take distinct values . For forecast group , let:
- be the number of cases;
- be the observed event frequency;
- be the event frequency across all cases.
Then
where
and
Reliability
Reliability measures agreement between forecast probabilities and observed frequencies. For example, among cases assigned probability , an event frequency near indicates good calibration at that forecast value. Lower reliability error is better.
Resolution
Resolution measures how much conditional event frequencies differ from the overall event frequency. Higher resolution is better: the forecasts identify groups with meaningfully different outcome probabilities.
A forecaster that always predicts the overall event rate can be calibrated but has no resolution. Calibration alone therefore does not establish strong predictive performance.
Uncertainty
Uncertainty depends only on the overall event frequency, not on the forecasts. It is greatest when the event rate is and decreases as the event becomes very rare or very common.
The displayed finite-sample identity is exact when groups contain identical forecast values. When continuously varying forecasts are placed into bins and represented by bin averages, the usual three-term calculation need not exactly reproduce the original score. Binning also affects estimated reliability and resolution, especially in small samples.
At the population level, if is the forecast probability and is a conditional expectation, the corresponding identity is
Brier skill score
The Brier skill score expresses improvement relative to a reference forecast:
provided the reference score is positive.
Its interpretation is:
- : perfect predictions;
- : the same score as the reference;
- : worse performance than the reference.
A common reference predicts a constant climatological or historical event frequency. If the reference is the event frequency calculated from the evaluation sample itself, its binary Brier score is exactly . An independently estimated historical frequency generally gives a different reference score.
Skill scores depend on the reference and evaluation sample. Their interpretation therefore requires the baseline to be specified. When the reference score is zero, the ratio is undefined.
Applications and comparison with other measures
In weather forecasting, the score evaluates probabilities of events such as precipitation or threshold exceedances. In machine learning, it evaluates probabilistic classifiers, including logistic regression models and neural networks.
Unlike classification accuracy or the F-score, it evaluates probabilities directly rather than labels obtained after choosing a threshold. Two models can make identical class predictions but receive different Brier scores because their confidence levels differ.
A common alternative is binary logarithmic loss, or cross-entropy:
Both are strictly proper, but they penalize errors differently. The Brier score is bounded; logarithmic loss becomes arbitrarily large when a forecast assigns a probability approaching zero to the outcome that occurs.
As a training objective, the Brier score is smooth and quadratic in the reported probabilities. Its behavior with respect to model parameters depends on the parameterization. For example, composing it with a logistic transformation can produce small gradients for highly confident incorrect predictions; cross-entropy has a different gradient structure.
Limitations and interpretation
Dependence on event frequency. Raw scores from datasets with different event rates are not directly comparable measures of predictive skill. A constant forecast for a very rare event can obtain a small score without distinguishing high-risk from low-risk cases.
Incomplete diagnostic information. A single value does not reveal whether errors arise mainly from poor calibration or weak resolution. Calibration plots and decompositions provide complementary information.
Sampling uncertainty. An empirical Brier score is an estimate of predictive performance. Cross-validation can support out-of-sample evaluation, while bootstrap methods can help quantify uncertainty when their sampling assumptions are appropriate. Dependence within a time series or clustered dataset requires an evaluation design that accounts for that dependence.
Dependence on the evaluation population. Changes in event prevalence or in the mix of cases can change the score even when the forecasting procedure remains unchanged.
Sensitivity to weighting. Unequal case weights change the population or distribution being evaluated. Outcome-dependent weights can also destroy propriety for the original event probability unless the scoring rule is adjusted appropriately.
No direct encoding of decision costs. The Brier score evaluates probability forecasts, not the consequences of acting on them. In decision theory, the preferred action also depends on utilities, costs, and available alternatives. A lower Brier score does not imply a better decision under every cost structure.