The continuous ranked probability score (CRPS) is a proper scoring rule for evaluating forecasts expressed as a probability distribution over a real-valued outcome. It measures the discrepancy between the forecast’s cumulative distribution and the cumulative distribution corresponding to the observed value. Lower scores are better. Unlike a score for a point prediction, CRPS assesses the entire predictive distribution, including its location and spread. It is widely used in weather forecasting and in other applications involving probabilistic predictions of continuous or ordered numerical quantities.
Definition
Let be the forecast cumulative distribution function and let be the observed outcome. The CRPS is
where is the indicator function.
The function , considered as a function of , is the cumulative distribution function of a distribution concentrated entirely at . Thus, CRPS is the integrated squared difference between the forecast CDF and the CDF of the realized observation.
An equivalent expression is
This form shows how the score penalizes forecast probability below an observation that turns out to be larger, and forecast probability above an observation that turns out to be smaller.
The score is nonnegative. It is zero precisely when the forecast puts all its probability at the observed value. A finite first absolute moment is a standard sufficient condition for finiteness and for the expectation-based formulas below.
Despite its name, CRPS does not require the forecast distribution to be continuous: it also applies to discrete and mixed distributions on the real line.
Equivalent representations
Expected absolute distances
Let and be independent random variables with distribution . If , then
Here denotes expectation. The first term measures the average absolute distance between a forecast draw and the observation. The second adjusts for the spread of the forecast distribution.
This adjustment distinguishes CRPS from simply averaging the absolute errors of forecast draws. Without it, minimizing the expected score would favor a point mass at a median of the outcome distribution rather than an honest representation of uncertainty.
Integrated threshold scores
For any threshold , the forecast probability of the event is , and its observed outcome is . Therefore,
is the Brier score for that binary event. CRPS integrates these binary scores over all thresholds.
This interpretation connects the evaluation of a numerical outcome to the evaluation of all threshold-exceedance or threshold-nonexceedance events associated with it.
Integrated quantile loss
Let denote the generalized quantile function, and define the quantile loss, or pinball loss, by
Then
CRPS can therefore be viewed either as an integral over thresholds or as an integral over quantile levels. The latter representation is useful when a forecasting model produces quantiles rather than a density or CDF.
Propriety and interpretation
A scoring rule is proper if reporting the true outcome distribution minimizes the expected score. It is strictly proper if that distribution is the unique minimizer.
Suppose the true distribution is , and and have finite first absolute moments. Then
The right-hand side is nonnegative and vanishes only when . Hence CRPS is strictly proper on this class of distributions.
The expected score can also be written as
The second term represents uncertainty inherent in the outcome distribution; it does not depend on the forecast. Even a correct, nondegenerate forecast therefore has a positive expected score.
Strict propriety is a statement about expected performance across outcomes. An incorrect forecast can outperform a correct one for a particular observation. Assessment consequently usually averages scores over many forecast–observation pairs.
Relation to point-forecast error
If the forecast assigns probability one to a point , then
CRPS thus extends absolute error from point forecasts to distributional forecasts.
It has the same physical units as the predicted quantity. For a transformed variable ,
Raw scores are therefore not directly comparable across quantities measured on different scales.
Computation
Finite ensembles
For an equally weighted ensemble , interpreted as an empirical predictive distribution,
This expression is exact for the empirical distribution. Direct evaluation of the pairwise term requires quadratic work, but sorting the ensemble gives
where are the sorted members. The resulting computation takes time.
For nonnegative weights summing to one,
A distinction arises when ensemble members are independent draws used to approximate an underlying distribution rather than the forecast distribution being evaluated. In that case, an unbiased estimator of the underlying CRPS is
This correction requires appropriate sampling assumptions; it is not automatically suitable for dependent or otherwise structured ensembles.
Normal forecasts
For a normal distribution with mean and standard deviation , define
If and are the standard normal CDF and probability density function, respectively, then
Closed-form expressions are also available for various other distribution families. Otherwise, the defining integral, quantile representation, or sample representation can be evaluated numerically.
Applications
CRPS is used to compare probabilistic weather forecasts, hydrological forecasts, economic predictions, and time-series models. It can also assess a posterior predictive distribution in Bayesian modeling.
For forecast–observation pairs , performance is commonly summarized by
A related continuous ranked probability skill score compares a model with a reference forecast:
provided the denominator is positive. Positive values indicate improvement over the reference; zero indicates equal mean scores; negative values indicate worse performance.
CRPS can also serve as a loss function for estimating predictive-distribution parameters. Minimizing empirical CRPS is an alternative to maximum likelihood estimation. Their objectives differ, so the resulting parameter estimates need not coincide.
Extensions and limitations
Threshold weighting
A threshold-weighted version is
where is fixed independently of the realized observation. It can emphasize thresholds associated with unusually high or low outcomes.
This construction remains proper under suitable integrability conditions. If the weight vanishes over a region, however, differences between distributions there may go undetected, so strict propriety for the entire distribution can be lost. Simply weighting ordinary scores according to the observed outcome does not generally preserve propriety.
Multivariate outcomes
Ordinary CRPS concerns a single real-valued variable. Scoring several variables separately assesses their marginal distributions but does not fully assess their dependence.
The energy score provides a multivariate extension by replacing absolute distances with Euclidean distances:
In one dimension, this is CRPS. In higher dimensions, sensitivity to dependence errors can be limited, making complementary assessments useful.
Diagnostic limitations
A single mean CRPS does not reveal whether forecast errors primarily concern location, dispersion, or tail behavior. Distinct forecasting systems can obtain similar scores while making different kinds of errors.
CRPS should also be distinguished from a standalone test of calibration. Its strict propriety rewards correct distributions in expectation, but a finite-sample score does not by itself establish that predicted probabilities agree with observed frequencies.
Because ordinary CRPS integrates discrepancies uniformly over thresholds, performance for rare extremes can be obscured by performance elsewhere. Moreover, aggregate scores depend on the evaluation cases: changes in forecast horizon, outcome variability, or case mix can change average CRPS without reflecting the same change in forecasting ability. Out-of-sample evaluation, including appropriately designed cross-validation, separates predictive assessment from in-sample fitting.