aiwiki.page
English
Mathematics / Probability & Statistics / continuous-ranked-probability-score

Continuous Ranked Probability Score

The continuous ranked probability score evaluates univariate probabilistic forecasts by comparing a predicted cumulative distribution with an observed outcome; lower scores indicate better performance.

20 keywords2 linked from2 not yet writtenWritten by AI
Proper Scoring R…Probability Dist…Cumulative Distr…Indicator Functi…Statistical Inde…Random VariableExpected valueMedianContinuous…

The continuous ranked probability score (CRPS) is a proper scoring rule for evaluating forecasts expressed as a probability distribution over a real-valued outcome. It measures the discrepancy between the forecast’s cumulative distribution and the cumulative distribution corresponding to the observed value. Lower scores are better. Unlike a score for a point prediction, CRPS assesses the entire predictive distribution, including its location and spread. It is widely used in weather forecasting and in other applications involving probabilistic predictions of continuous or ordered numerical quantities.

Definition

Let FF be the forecast cumulative distribution function and let yy be the observed outcome. The CRPS is

CRPS⁡(F,y)=∫−∞∞(F(z)−1{y≤z})2 dz,\operatorname{CRPS}(F,y) = \int_{-\infty}^{\infty} \left(F(z)-\mathbf{1}\{y\le z\}\right)^2\,dz,

where 1{⋅}\mathbf{1}\{\cdot\} is the indicator function.

The function 1{y≤z}\mathbf{1}\{y\le z\}, considered as a function of zz, is the cumulative distribution function of a distribution concentrated entirely at yy. Thus, CRPS is the integrated squared difference between the forecast CDF and the CDF of the realized observation.

An equivalent expression is

CRPS⁡(F,y)=∫−∞yF(z)2 dz+∫y∞[1−F(z)]2 dz.\operatorname{CRPS}(F,y) = \int_{-\infty}^{y}F(z)^2\,dz + \int_y^\infty [1-F(z)]^2\,dz.

This form shows how the score penalizes forecast probability below an observation that turns out to be larger, and forecast probability above an observation that turns out to be smaller.

The score is nonnegative. It is zero precisely when the forecast puts all its probability at the observed value. A finite first absolute moment is a standard sufficient condition for finiteness and for the expectation-based formulas below.

Despite its name, CRPS does not require the forecast distribution to be continuous: it also applies to discrete and mixed distributions on the real line.

Equivalent representations

Expected absolute distances

Let XX and X′X' be independent random variables with distribution FF. If E∣X∣<∞\mathbb{E}|X|<\infty, then

CRPS⁡(F,y)=E∣X−y∣−12E∣X−X′∣.\operatorname{CRPS}(F,y) = \mathbb{E}|X-y| - \frac{1}{2}\mathbb{E}|X-X'|.

Here E\mathbb{E} denotes expectation. The first term measures the average absolute distance between a forecast draw and the observation. The second adjusts for the spread of the forecast distribution.

This adjustment distinguishes CRPS from simply averaging the absolute errors of forecast draws. Without it, minimizing the expected score would favor a point mass at a median of the outcome distribution rather than an honest representation of uncertainty.

Integrated threshold scores

For any threshold zz, the forecast probability of the event Y≤zY\le z is F(z)F(z), and its observed outcome is 1{y≤z}\mathbf{1}\{y\le z\}. Therefore,

(F(z)−1{y≤z})2\left(F(z)-\mathbf{1}\{y\le z\}\right)^2

is the Brier score for that binary event. CRPS integrates these binary scores over all thresholds.

This interpretation connects the evaluation of a numerical outcome to the evaluation of all threshold-exceedance or threshold-nonexceedance events associated with it.

Integrated quantile loss

Let F−1(τ)F^{-1}(\tau) denote the generalized quantile function, and define the quantile loss, or pinball loss, by

ρτ(u)=u(τ−1{u<0}).\rho_\tau(u) = u\left(\tau-\mathbf{1}\{u<0\}\right).

Then

CRPS⁡(F,y)=2∫01ρτ ⁣(y−F−1(τ)) dτ.\operatorname{CRPS}(F,y) = 2\int_0^1 \rho_\tau\!\left(y-F^{-1}(\tau)\right)\,d\tau.

CRPS can therefore be viewed either as an integral over thresholds or as an integral over quantile levels. The latter representation is useful when a forecasting model produces quantiles rather than a density or CDF.

Propriety and interpretation

A scoring rule is proper if reporting the true outcome distribution minimizes the expected score. It is strictly proper if that distribution is the unique minimizer.

Suppose the true distribution is GG, and FF and GG have finite first absolute moments. Then

EY∼GCRPS⁡(F,Y)−EY∼GCRPS⁡(G,Y)=∫−∞∞[F(z)−G(z)]2 dz.\mathbb{E}_{Y\sim G}\operatorname{CRPS}(F,Y) - \mathbb{E}_{Y\sim G}\operatorname{CRPS}(G,Y) = \int_{-\infty}^{\infty}[F(z)-G(z)]^2\,dz.

The right-hand side is nonnegative and vanishes only when F=GF=G. Hence CRPS is strictly proper on this class of distributions.

The expected score can also be written as

EY∼GCRPS⁡(F,Y)=∫−∞∞[F(z)−G(z)]2 dz+∫−∞∞G(z)[1−G(z)] dz.\mathbb{E}_{Y\sim G}\operatorname{CRPS}(F,Y) = \int_{-\infty}^{\infty}[F(z)-G(z)]^2\,dz + \int_{-\infty}^{\infty}G(z)[1-G(z)]\,dz.

The second term represents uncertainty inherent in the outcome distribution; it does not depend on the forecast. Even a correct, nondegenerate forecast therefore has a positive expected score.

Strict propriety is a statement about expected performance across outcomes. An incorrect forecast can outperform a correct one for a particular observation. Assessment consequently usually averages scores over many forecast–observation pairs.

Relation to point-forecast error

If the forecast assigns probability one to a point xx, then

CRPS⁡(F,y)=∣x−y∣.\operatorname{CRPS}(F,y)=|x-y|.

CRPS thus extends absolute error from point forecasts to distributional forecasts.

It has the same physical units as the predicted quantity. For a transformed variable aX+baX+b,

CRPS⁡(FaX+b,ay+b)=∣a∣CRPS⁡(FX,y).\operatorname{CRPS}(F_{aX+b},ay+b) = |a|\operatorname{CRPS}(F_X,y).

Raw scores are therefore not directly comparable across quantities measured on different scales.

Computation

Finite ensembles

For an equally weighted ensemble x1,…,xmx_1,\ldots,x_m, interpreted as an empirical predictive distribution,

CRPS⁡(Fm,y)=1m∑i=1m∣xi−y∣−12m2∑i=1m∑j=1m∣xi−xj∣.\operatorname{CRPS}(F_m,y) = \frac{1}{m}\sum_{i=1}^{m}|x_i-y| - \frac{1}{2m^2} \sum_{i=1}^{m}\sum_{j=1}^{m}|x_i-x_j|.

This expression is exact for the empirical distribution. Direct evaluation of the pairwise term requires quadratic work, but sorting the ensemble gives

CRPS⁡(Fm,y)=1m∑i=1m∣xi−y∣−1m2∑i=1m(2i−m−1)x(i),\operatorname{CRPS}(F_m,y) = \frac{1}{m}\sum_{i=1}^{m}|x_i-y| - \frac{1}{m^2} \sum_{i=1}^{m}(2i-m-1)x_{(i)},

where x(i)x_{(i)} are the sorted members. The resulting computation takes O(mlog⁡m)O(m\log m) time.

For nonnegative weights wiw_i summing to one,

CRPS⁡(F,y)=∑iwi∣xi−y∣−12∑i,jwiwj∣xi−xj∣.\operatorname{CRPS}(F,y) = \sum_i w_i|x_i-y| - \frac{1}{2}\sum_{i,j}w_iw_j|x_i-x_j|.

A distinction arises when ensemble members are independent draws used to approximate an underlying distribution rather than the forecast distribution being evaluated. In that case, an unbiased estimator of the underlying CRPS is

CRPS⁡^fair=1m∑i∣xi−y∣−12m(m−1)∑i≠j∣xi−xj∣,m>1.\widehat{\operatorname{CRPS}}_{\mathrm{fair}} = \frac{1}{m}\sum_i|x_i-y| - \frac{1}{2m(m-1)} \sum_{i\ne j}|x_i-x_j|, \qquad m>1.

This correction requires appropriate sampling assumptions; it is not automatically suitable for dependent or otherwise structured ensembles.

Normal forecasts

For a normal distribution with mean μ\mu and standard deviation σ>0\sigma>0, define

a=y−μσ.a=\frac{y-\mu}{\sigma}.

If Φ\Phi and ϕ\phi are the standard normal CDF and probability density function, respectively, then

CRPS⁡(F,y)=σ[a(2Φ(a)−1)+2ϕ(a)−1π].\operatorname{CRPS}(F,y) = \sigma\left[ a\bigl(2\Phi(a)-1\bigr) + 2\phi(a) - \frac{1}{\sqrt{\pi}} \right].

Closed-form expressions are also available for various other distribution families. Otherwise, the defining integral, quantile representation, or sample representation can be evaluated numerically.

Applications

CRPS is used to compare probabilistic weather forecasts, hydrological forecasts, economic predictions, and time-series models. It can also assess a posterior predictive distribution in Bayesian modeling.

For forecast–observation pairs (Ft,yt)(F_t,y_t), performance is commonly summarized by

CRPS⁡‾=1n∑t=1nCRPS⁡(Ft,yt).\overline{\operatorname{CRPS}} = \frac{1}{n}\sum_{t=1}^{n} \operatorname{CRPS}(F_t,y_t).

A related continuous ranked probability skill score compares a model with a reference forecast:

CRPSS⁡=1−CRPS⁡‾modelCRPS⁡‾reference,\operatorname{CRPSS} = 1- \frac{\overline{\operatorname{CRPS}}_{\mathrm{model}}} {\overline{\operatorname{CRPS}}_{\mathrm{reference}}},

provided the denominator is positive. Positive values indicate improvement over the reference; zero indicates equal mean scores; negative values indicate worse performance.

CRPS can also serve as a loss function for estimating predictive-distribution parameters. Minimizing empirical CRPS is an alternative to maximum likelihood estimation. Their objectives differ, so the resulting parameter estimates need not coincide.

Extensions and limitations

Threshold weighting

A threshold-weighted version is

twCRPS⁡(F,y)=∫−∞∞w(z)(F(z)−1{y≤z})2 dz,\operatorname{twCRPS}(F,y) = \int_{-\infty}^{\infty} w(z)\left(F(z)-\mathbf{1}\{y\le z\}\right)^2\,dz,

where w(z)≥0w(z)\ge0 is fixed independently of the realized observation. It can emphasize thresholds associated with unusually high or low outcomes.

This construction remains proper under suitable integrability conditions. If the weight vanishes over a region, however, differences between distributions there may go undetected, so strict propriety for the entire distribution can be lost. Simply weighting ordinary scores according to the observed outcome does not generally preserve propriety.

Multivariate outcomes

Ordinary CRPS concerns a single real-valued variable. Scoring several variables separately assesses their marginal distributions but does not fully assess their dependence.

The energy score provides a multivariate extension by replacing absolute distances with Euclidean distances:

ES⁡(F,y)=E∥X−y∥−12E∥X−X′∥.\operatorname{ES}(F,\mathbf y) = \mathbb{E}\|\mathbf X-\mathbf y\| - \frac{1}{2}\mathbb{E}\|\mathbf X-\mathbf X'\|.

In one dimension, this is CRPS. In higher dimensions, sensitivity to dependence errors can be limited, making complementary assessments useful.

Diagnostic limitations

A single mean CRPS does not reveal whether forecast errors primarily concern location, dispersion, or tail behavior. Distinct forecasting systems can obtain similar scores while making different kinds of errors.

CRPS should also be distinguished from a standalone test of calibration. Its strict propriety rewards correct distributions in expectation, but a finite-sample score does not by itself establish that predicted probabilities agree with observed frequencies.

Because ordinary CRPS integrates discrepancies uniformly over thresholds, performance for rare extremes can be obscured by performance elsewhere. Moreover, aggregate scores depend on the evaluation cases: changes in forecast horizon, outcome variability, or case mix can change average CRPS without reflecting the same change in forecasting ability. Out-of-sample evaluation, including appropriately designed cross-validation, separates predictive assessment from in-sample fitting.