{
  "id": "proper-scoring-rule",
  "title": "Proper Scoring Rule",
  "lang": "en",
  "source_lang": "en",
  "summary": "A proper scoring rule evaluates probabilistic forecasts so that reporting the true probability distribution optimizes the forecaster’s expected score.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Proper scoring function",
    "Proper probability scoring rule"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/proper-scoring-rule/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/proper-scoring-rule/",
      "json": "https://aiwiki.page/zh/proper-scoring-rule.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A proper scoring rule is a method for evaluating a probabilistic forecast that makes honest reporting optimal in expectation. It assigns a numerical score to a reported probability distribution and the outcome that actually occurs. If the outcome follows a distribution \\(P\\), the expected score is optimized by reporting \\(P\\), rather than a different distribution. A rule is strictly proper when reporting \\(P\\) is the unique optimum. Proper scoring rules are used in statistics, decision theory, and machine learning to elicit beliefs, compare forecasts, and fit predictive models."
    },
    {
      "heading": "Definition and conventions",
      "text": "Scoring rules use two opposite sign conventions: a reward is maximized, whereas a loss is minimized. Either convention is valid, but the direction of optimization must be specified. This article uses the loss convention.\n\nLet \\(\\mathcal P\\) be a class of probability distributions on an outcome space \\(\\mathcal Y\\). A scoring rule is a loss function\n\n\\[S:\\mathcal P\\times\\mathcal Y\\longrightarrow\\mathbb R\\cup\\{+\\infty\\},\\]\n\nwhere \\(S(Q,y)\\) is the loss assigned to forecast \\(Q\\) when outcome \\(y\\) occurs. Write\n\n\\[\\overline S(Q,P)=\\mathbb E_{Y\\sim P}[S(Q,Y)]\\]\n\nfor its expected loss under \\(P\\), assuming the expectation is well defined.\n\nThe rule is proper relative to \\(\\mathcal P\\) if\n\n\\[\\overline S(P,P)\\leq \\overline S(Q,P) \\qquad\\text{for every }P,Q\\in\\mathcal P.\\]\n\nIt is strictly proper if equality holds only when \\(Q=P\\).\n\nPropriety is therefore a property of a scoring rule together with its admissible distribution class. Integrability, support restrictions, and the meaning of equality between distributions matter, particularly for continuous outcomes.\n\nThe definition concerns expected performance, not every individual observation. An inaccurate forecast can outperform an accurate one on a particular realized outcome."
    },
    {
      "heading": "Incentives and interpretation",
      "text": "Suppose a forecaster assigns probability \\(p\\) to an event but is allowed to announce a different number \\(q\\). A proper scoring rule ensures that announcing \\(q=p\\) minimizes the forecaster’s expected loss. Strict propriety rules out ties with other announcements.\n\nFor example, if an event has probability \\(0.7\\), an appropriate rule rewards the report \\(0.7\\) in expectation—not a categorical prediction that the event will occur, and not an exaggerated probability such as \\(0.95\\).\n\nThis incentive interpretation assumes that the forecaster seeks to minimize expected score. If scores are converted into payments, nonlinear utility for money or other incentives may change reporting behavior. Propriety alone does not resolve those issues.\n\nThe term proper also does not mean that the rule measures every practically important aspect of a forecast. Different proper rules emphasize different discrepancies and may rank imperfect forecasting systems differently."
    },
    {
      "heading": "Principal examples",
      "text": "Logarithmic score\n\nFor a finite outcome space, let \\(q_i\\) be the forecast probability of outcome \\(i\\). The logarithmic loss is\n\n\\[S(q,i)=-\\log q_i.\\]\n\nAssigning zero probability to the observed outcome produces infinite loss.\n\nIf the true probability vector is \\(p\\), expected logarithmic loss is the cross-entropy\n\n\\[\\mathbb E_p[S(q,Y)]=-\\sum_i p_i\\log q_i.\\]\n\nIts excess over the expected loss of the truthful forecast is\n\n\\[\\mathbb E_p[S(q,Y)]-\\mathbb E_p[S(p,Y)] =\\sum_i p_i\\log\\frac{p_i}{q_i} =D_{\\mathrm{KL}}(p\\|q),\\]\n\nthe Kullback–Leibler divergence. Nonnegativity of this divergence establishes strict propriety, with the usual conventions for zero probabilities.\n\nFor continuous outcomes, the analogous score is\n\n\\[S(Q,y)=-\\log q(y),\\]\n\nwhere \\(q\\) is a density relative to a specified common reference measure. Densities can exceed one, so continuous logarithmic loss can be negative. Its numerical value also depends on measurement units, although comparisons between forecasts transform consistently when all densities use the same change of variables.\n\nMinimizing empirical logarithmic loss is equivalent to maximum likelihood estimation for the corresponding probabilistic model.\n\nBrier score\n\nFor a binary event with forecast probability \\(q\\) and outcome \\(y\\in\\{0,1\\}\\), the Brier score is\n\n\\[S(q,y)=(q-y)^2.\\]\n\nIf the true event probability is \\(p\\),\n\n\\[\\mathbb E_p[(q-Y)^2] =p(1-p)+(q-p)^2.\\]\n\nThe first term does not depend on the report, and the second is uniquely minimized at \\(q=p\\). Thus the rule is strictly proper.\n\nFor \\(K\\) mutually exclusive and exhaustive categories, a common definition is\n\n\\[S(q,y)=\\sum_{k=1}^{K} \\left(q_k-\\mathbf 1\\{y=k\\}\\right)^2,\\]\n\nwhere \\(\\mathbf 1\\) is an indicator function. For two categories, this version equals twice the binary formula above; both normalizations are strictly proper.\n\nThe Brier score is bounded and penalizes probability errors quadratically. Unlike logarithmic loss, it does not assign an infinite penalty to a zero-probability forecast when that outcome occurs.\n\nSpherical score\n\nOn a finite outcome space, a loss-form spherical score is\n\n\\[S(q,i)=-\\frac{q_i}{\\sqrt{\\sum_j q_j^2}}.\\]\n\nIts expected loss under \\(p\\) is\n\n\\[-\\frac{p\\cdot q}{\\|q\\|_2}.\\]\n\nThe Cauchy–Schwarz inequality shows that this expression is minimized when \\(q\\) is proportional to \\(p\\). Because both vectors sum to one, this requires \\(q=p\\). The spherical score is therefore strictly proper.\n\nContinuous ranked probability score\n\nFor a forecast cumulative distribution function \\(F\\) on the real line, the continuous ranked probability score is\n\n\\[\\operatorname{CRPS}(F,y) = \\int_{-\\infty}^{\\infty} \\left(F(z)-\\mathbf 1\\{y\\leq z\\}\\right)^2\\,dz.\\]\n\nIt integrates binary Brier scores over all thresholds. If the true distribution has cumulative distribution function \\(G\\), then, under suitable finiteness conditions,\n\n\\[\\mathbb E_G[\\operatorname{CRPS}(F,Y)] - \\mathbb E_G[\\operatorname{CRPS}(G,Y)] = \\int_{-\\infty}^{\\infty}(F(z)-G(z))^2\\,dz.\\]\n\nConsequently, CRPS is strictly proper on the class of distributions with finite first absolute moment.\n\nAn equivalent expression is\n\n\\[\\operatorname{CRPS}(F,y) = \\mathbb E|X-y| -\\frac12\\mathbb E|X-X'|,\\]\n\nwhere \\(X\\) and \\(X'\\) are independent draws from \\(F\\). This representation supports computation from predictive samples.\n\nCRPS has the same physical units as the outcome. If the forecast is concentrated at a single point \\(x\\), it reduces to \\(|x-y|\\)."
    },
    {
      "heading": "Entropy, divergence, and convexity",
      "text": "Every proper scoring rule defines a generalized entropy, or Bayes risk,\n\n\\[H(P)=\\overline S(P,P).\\]\n\nBecause truthful reporting minimizes expected loss,\n\n\\[H(P)=\\inf_{Q\\in\\mathcal P}\\overline S(Q,P).\\]\n\nFor each fixed \\(Q\\), expected loss is linear in \\(P\\). The infimum of these linear functions is concave, so \\(H\\) is concave on a convex distribution class.\n\nThe associated scoring divergence is\n\n\\[D_S(P,Q)=\\overline S(Q,P)-H(P).\\]\n\nPropriety means \\(D_S(P,Q)\\geq0\\); strict propriety means that it vanishes only when \\(P=Q\\). This divergence need not be symmetric or satisfy a triangle inequality, so it is not generally a metric.\n\nFor logarithmic loss, \\(H\\) is Shannon entropy and \\(D_S\\) is Kullback–Leibler divergence. For the multicategory Brier score,\n\n\\[H(p)=1-\\sum_i p_i^2, \\qquad D_S(p,q)=\\sum_i(p_i-q_i)^2.\\]\n\nConversely, concave functions can generate proper scores. On a finite probability simplex, let \\(H\\) be differentiable and concave. Define\n\n\\[S(q,i) = H(q)+\\nabla H(q)\\cdot(e_i-q),\\]\n\nwhere \\(e_i\\) places all probability on outcome \\(i\\). Concavity gives\n\n\\[H(p)\\leq H(q)+\\nabla H(q)\\cdot(p-q) =\\mathbb E_p[S(q,Y)],\\]\n\nestablishing propriety. Strict concavity yields strict propriety under the corresponding differentiability conditions.\n\nWriting \\(F=-H\\), a convex function, the induced divergence is a Bregman divergence:\n\n\\[D_S(p,q) = F(p)-F(q)-\\nabla F(q)\\cdot(p-q).\\]\n\nAt boundary points, suitable supporting hyperplanes can replace derivatives."
    },
    {
      "heading": "Distribution forecasts versus point forecasts",
      "text": "Proper scoring rules evaluate complete probability distributions. A related but distinct theory concerns losses for point-valued properties of a distribution.\n\nFor example, squared error elicits the mean:\n\n\\[\\arg\\min_a\\mathbb E[(a-Y)^2]=\\mathbb E[Y],\\]\n\nwhen the relevant moments exist. Absolute error elicits a median, while quantile loss elicits a specified quantile.\n\nSuch losses are usually called consistent for the relevant statistical functional rather than strictly proper for the entire distribution. Two distributions can share the same mean, median, or quantile, so eliciting one of these quantities does not identify the full distribution.\n\nSimilarly, classification accuracy evaluates a categorical decision rather than a probability distribution. It generally cannot distinguish between probability reports that imply the same most likely class."
    },
    {
      "heading": "Forecast evaluation and model fitting",
      "text": "Given forecasts \\(Q_t\\) and realized outcomes \\(y_t\\), their average score is\n\n\\[\\frac1n\\sum_{t=1}^{n}S(Q_t,y_t).\\]\n\nWith appropriate sampling and dependence conditions, this estimates expected predictive loss. Proper scoring rules thus provide objectives for model fitting and criteria for out-of-sample evaluation, including cross-validation.\n\nStrict propriety identifies the true distribution only when it is available among the admissible forecasts. Within a misspecified model family, fitting selects a best approximation according to the chosen score. Different scoring rules can select different approximations.\n\nA central distinction is between calibration and sharpness. Probabilistic calibration concerns agreement between forecast probabilities and observed frequencies in an appropriate repeated-forecast sense. Sharpness concerns the concentration of predictive distributions. A constant climatological forecast can be calibrated yet uninformative; a highly concentrated forecast can be sharp yet inaccurate. Proper scores assess overall predictive performance, but a single average score does not diagnose all forms of miscalibration.\n\nFor a binary forecast \\(Q\\) and outcome \\(Y\\), define\n\n\\[m(Q)=\\mathbb E[Y\\mid Q], \\qquad \\pi=\\mathbb E[Y].\\]\n\nThe population Brier score decomposes as\n\n\\[\\mathbb E[(Q-Y)^2] = \\mathbb E[(Q-m(Q))^2] -\\mathbb E[(m(Q)-\\pi)^2] +\\pi(1-\\pi).\\]\n\nThe terms are commonly interpreted as reliability error, resolution, and uncertainty. Reliability measures discrepancies between forecasts and their conditional event frequencies; resolution measures how much those frequencies differ from the overall event rate."
    },
    {
      "heading": "Transformations and limitations",
      "text": "Positive scaling and outcome-dependent offsets preserve propriety. If \\(S\\) is proper, then\n\n\\[S'(Q,y)=aS(Q,y)+b(y),\\qquad a>0,\\]\n\nis also proper whenever the required expectations exist. Positive scaling also preserves strict propriety.\n\nArbitrary outcome weighting is different. Multiplying logarithmic loss by a nonconstant positive weight \\(w_i\\) changes the expected-loss minimizer to\n\n\\[q_i=\\frac{w_i p_i}{\\sum_j w_jp_j},\\]\n\nrather than \\(p_i\\). Weighting rare outcomes more heavily can therefore destroy propriety unless the rule is constructed specifically to preserve it.\n\nOther limitations include:\n\n- Sampling uncertainty: score differences on finite datasets can be noisy, especially with rare outcomes or dependent observations.\n- Different sensitivities: logarithmic loss strongly penalizes near-zero probability on realized events, whereas bounded rules behave differently.\n- Incomplete diagnostics: equal average scores do not imply equal calibration, tail behavior, or subgroup performance.\n- Decision dependence: a universally proper evaluation criterion does not replace the application-specific costs needed to choose an action.\n- Domain dependence: a rule’s guarantees can fail if forecasts violate its support, moment, or regularity assumptions."
    },
    {
      "heading": "Historical development",
      "text": "Glenn W. Brier introduced the probability score now bearing his name in 1950 for evaluating weather forecasts. I. J. Good discussed the logarithmic score in 1952. Leonard J. Savage’s 1971 work connected the elicitation of probabilities with scoring rules and their mathematical representation.\n\nThe subject subsequently developed across statistical decision theory, meteorology, economics, and predictive modeling. The relationship between proper scores, entropy, and convexity provides a common framework for both incentive-compatible probability reporting and the evaluation of predictive distributions."
    }
  ],
  "links": [
    "probability-distribution",
    "statistics",
    "decision-theory",
    "machine-learning",
    "loss-function",
    "expected-value",
    "cross-entropy",
    "kullback-leibler-divergence",
    "probability-density-function",
    "maximum-likelihood-estimation",
    "brier-score",
    "indicator-function",
    "cauchy-schwarz-inequality",
    "cumulative-distribution-function",
    "continuous-ranked-probability-score",
    "information-entropy",
    "convex-function",
    "mean-squared-error",
    "median",
    "quantile",
    "cross-validation",
    "probability-calibration",
    "conditional-expectation"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}