A proper scoring rule is a method for evaluating a probabilistic forecast that makes honest reporting optimal in expectation. It assigns a numerical score to a reported probability distribution and the outcome that actually occurs. If the outcome follows a distribution , the expected score is optimized by reporting , rather than a different distribution. A rule is strictly proper when reporting is the unique optimum. Proper scoring rules are used in statistics, decision theory, and machine learning to elicit beliefs, compare forecasts, and fit predictive models.
Definition and conventions
Scoring rules use two opposite sign conventions: a reward is maximized, whereas a loss is minimized. Either convention is valid, but the direction of optimization must be specified. This article uses the loss convention.
Let be a class of probability distributions on an outcome space . A scoring rule is a loss function
where is the loss assigned to forecast when outcome occurs. Write
for its expected loss under , assuming the expectation is well defined.
The rule is proper relative to if
It is strictly proper if equality holds only when .
Propriety is therefore a property of a scoring rule together with its admissible distribution class. Integrability, support restrictions, and the meaning of equality between distributions matter, particularly for continuous outcomes.
The definition concerns expected performance, not every individual observation. An inaccurate forecast can outperform an accurate one on a particular realized outcome.
Incentives and interpretation
Suppose a forecaster assigns probability to an event but is allowed to announce a different number . A proper scoring rule ensures that announcing minimizes the forecaster’s expected loss. Strict propriety rules out ties with other announcements.
For example, if an event has probability , an appropriate rule rewards the report in expectation—not a categorical prediction that the event will occur, and not an exaggerated probability such as .
This incentive interpretation assumes that the forecaster seeks to minimize expected score. If scores are converted into payments, nonlinear utility for money or other incentives may change reporting behavior. Propriety alone does not resolve those issues.
The term proper also does not mean that the rule measures every practically important aspect of a forecast. Different proper rules emphasize different discrepancies and may rank imperfect forecasting systems differently.
Principal examples
Logarithmic score
For a finite outcome space, let be the forecast probability of outcome . The logarithmic loss is
Assigning zero probability to the observed outcome produces infinite loss.
If the true probability vector is , expected logarithmic loss is the cross-entropy
Its excess over the expected loss of the truthful forecast is
the Kullback–Leibler divergence. Nonnegativity of this divergence establishes strict propriety, with the usual conventions for zero probabilities.
For continuous outcomes, the analogous score is
where is a density relative to a specified common reference measure. Densities can exceed one, so continuous logarithmic loss can be negative. Its numerical value also depends on measurement units, although comparisons between forecasts transform consistently when all densities use the same change of variables.
Minimizing empirical logarithmic loss is equivalent to maximum likelihood estimation for the corresponding probabilistic model.
Brier score
For a binary event with forecast probability and outcome , the Brier score is
If the true event probability is ,
The first term does not depend on the report, and the second is uniquely minimized at . Thus the rule is strictly proper.
For mutually exclusive and exhaustive categories, a common definition is
where is an indicator function. For two categories, this version equals twice the binary formula above; both normalizations are strictly proper.
The Brier score is bounded and penalizes probability errors quadratically. Unlike logarithmic loss, it does not assign an infinite penalty to a zero-probability forecast when that outcome occurs.
Spherical score
On a finite outcome space, a loss-form spherical score is
Its expected loss under is
The Cauchy–Schwarz inequality shows that this expression is minimized when is proportional to . Because both vectors sum to one, this requires . The spherical score is therefore strictly proper.
Continuous ranked probability score
For a forecast cumulative distribution function on the real line, the continuous ranked probability score is
It integrates binary Brier scores over all thresholds. If the true distribution has cumulative distribution function , then, under suitable finiteness conditions,
Consequently, CRPS is strictly proper on the class of distributions with finite first absolute moment.
An equivalent expression is
where and are independent draws from . This representation supports computation from predictive samples.
CRPS has the same physical units as the outcome. If the forecast is concentrated at a single point , it reduces to .
Entropy, divergence, and convexity
Every proper scoring rule defines a generalized entropy, or Bayes risk,
Because truthful reporting minimizes expected loss,
For each fixed , expected loss is linear in . The infimum of these linear functions is concave, so is concave on a convex distribution class.
The associated scoring divergence is
Propriety means ; strict propriety means that it vanishes only when . This divergence need not be symmetric or satisfy a triangle inequality, so it is not generally a metric.
For logarithmic loss, is Shannon entropy and is Kullback–Leibler divergence. For the multicategory Brier score,
Conversely, concave functions can generate proper scores. On a finite probability simplex, let be differentiable and concave. Define
where places all probability on outcome . Concavity gives
establishing propriety. Strict concavity yields strict propriety under the corresponding differentiability conditions.
Writing , a convex function, the induced divergence is a Bregman divergence:
At boundary points, suitable supporting hyperplanes can replace derivatives.
Distribution forecasts versus point forecasts
Proper scoring rules evaluate complete probability distributions. A related but distinct theory concerns losses for point-valued properties of a distribution.
For example, squared error elicits the mean:
when the relevant moments exist. Absolute error elicits a median, while quantile loss elicits a specified quantile.
Such losses are usually called consistent for the relevant statistical functional rather than strictly proper for the entire distribution. Two distributions can share the same mean, median, or quantile, so eliciting one of these quantities does not identify the full distribution.
Similarly, classification accuracy evaluates a categorical decision rather than a probability distribution. It generally cannot distinguish between probability reports that imply the same most likely class.
Forecast evaluation and model fitting
Given forecasts and realized outcomes , their average score is
With appropriate sampling and dependence conditions, this estimates expected predictive loss. Proper scoring rules thus provide objectives for model fitting and criteria for out-of-sample evaluation, including cross-validation.
Strict propriety identifies the true distribution only when it is available among the admissible forecasts. Within a misspecified model family, fitting selects a best approximation according to the chosen score. Different scoring rules can select different approximations.
A central distinction is between calibration and sharpness. Probabilistic calibration concerns agreement between forecast probabilities and observed frequencies in an appropriate repeated-forecast sense. Sharpness concerns the concentration of predictive distributions. A constant climatological forecast can be calibrated yet uninformative; a highly concentrated forecast can be sharp yet inaccurate. Proper scores assess overall predictive performance, but a single average score does not diagnose all forms of miscalibration.
For a binary forecast and outcome , define
The population Brier score decomposes as
The terms are commonly interpreted as reliability error, resolution, and uncertainty. Reliability measures discrepancies between forecasts and their conditional event frequencies; resolution measures how much those frequencies differ from the overall event rate.
Transformations and limitations
Positive scaling and outcome-dependent offsets preserve propriety. If is proper, then
is also proper whenever the required expectations exist. Positive scaling also preserves strict propriety.
Arbitrary outcome weighting is different. Multiplying logarithmic loss by a nonconstant positive weight changes the expected-loss minimizer to
rather than . Weighting rare outcomes more heavily can therefore destroy propriety unless the rule is constructed specifically to preserve it.
Other limitations include:
- Sampling uncertainty: score differences on finite datasets can be noisy, especially with rare outcomes or dependent observations.
- Different sensitivities: logarithmic loss strongly penalizes near-zero probability on realized events, whereas bounded rules behave differently.
- Incomplete diagnostics: equal average scores do not imply equal calibration, tail behavior, or subgroup performance.
- Decision dependence: a universally proper evaluation criterion does not replace the application-specific costs needed to choose an action.
- Domain dependence: a rule’s guarantees can fail if forecasts violate its support, moment, or regularity assumptions.
Historical development
Glenn W. Brier introduced the probability score now bearing his name in 1950 for evaluating weather forecasts. I. J. Good discussed the logarithmic score in 1952. Leonard J. Savage’s 1971 work connected the elicitation of probabilities with scoring rules and their mathematical representation.
The subject subsequently developed across statistical decision theory, meteorology, economics, and predictive modeling. The relationship between proper scores, entropy, and convexity provides a common framework for both incentive-compatible probability reporting and the evaluation of predictive distributions.