{
  "id": "mean-reciprocal-rank",
  "title": "Mean reciprocal rank",
  "lang": "en",
  "source_lang": "en",
  "summary": "Mean reciprocal rank is a ranking evaluation metric that averages the inverse position of the first relevant result across queries or other evaluation cases.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "MRR",
    "Mean reciprocal ranking"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/mean-reciprocal-rank/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/mean-reciprocal-rank/",
      "json": "https://aiwiki.page/zh/mean-reciprocal-rank.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Mean reciprocal rank (MRR) is an evaluation metric used in information retrieval, recommender systems, and other tasks that produce ordered lists of candidates. For each query or evaluation case, it takes the reciprocal of the rank of the first relevant result, then averages these values across cases. MRR rewards systems that place a relevant result near the top of the list. It is especially suited to tasks in which finding one satisfactory answer matters more than retrieving every relevant item."
    },
    {
      "heading": "Definition",
      "text": "Let \\(Q\\) be a nonempty set of evaluation queries. For each query \\(q\\), let \\(r_q\\) be the position of its first relevant result, with positions counted from 1. The reciprocal rank for that query is\n\n\\[\\operatorname{RR}(q)= \\begin{cases} \\dfrac{1}{r_q}, & \\text{if a relevant result is retrieved},\\\\[6pt] 0, & \\text{otherwise}. \\end{cases}\\]\n\nMean reciprocal rank is\n\n\\[\\operatorname{MRR} = \\frac{1}{|Q|} \\sum_{q\\in Q}\\operatorname{RR}(q).\\]\n\nA query can represent a search request, a question, a recommendation context, or another case with an ordered candidate list and a defined relevance criterion.\n\nMRR lies between 0 and 1. A score of 1 means that every evaluation case has a relevant result in first position. A score of 0 means that no relevant result appears in any evaluated list. Higher scores indicate better performance under this metric.\n\nThe standard definition gives each query equal weight, regardless of the number of candidates or relevant results associated with it. Weighted variants are possible, but their weighting scheme must be stated explicitly."
    },
    {
      "heading": "Example",
      "text": "Suppose a system is evaluated on four queries:\n\nQuery | Position of first relevant result | Reciprocal rank\nA | 1 | \\(1\\)\nB | 2 | \\(1/2\\)\nC | 4 | \\(1/4\\)\nD | No relevant result retrieved | \\(0\\)\n\nIts MRR is\n\n\\[\\operatorname{MRR} = \\frac{1+\\frac12+\\frac14+0}{4} = 0.4375.\\]\n\nOnly the first relevant result contributes to each query’s score. If query B also has relevant results at positions 3 and 5, its reciprocal rank remains \\(1/2\\).\n\nMRR is not the reciprocal of the mean rank. For first-relevant ranks of 1 and 3, MRR is\n\n\\[\\frac{1+\\frac13}{2}=\\frac23,\\]\n\nwhereas the reciprocal of their mean rank is\n\n\\[\\frac{1}{(1+3)/2}=\\frac12.\\]\n\nConsequently, taking the reciprocal of an MRR score does not generally recover the average position of the first relevant result."
    },
    {
      "heading": "Truncation: MRR at a cutoff",
      "text": "Many evaluations inspect only the first \\(k\\) results. The corresponding metric, usually written MRR@\\(k\\), assigns zero to a query when its first relevant result falls below the cutoff:\n\n\\[\\operatorname{RR}@k(q)= \\begin{cases} \\dfrac{1}{r_q}, & r_q\\leq k,\\\\[6pt] 0, & \\text{otherwise}. \\end{cases}\\]\n\n\\[\\operatorname{MRR}@k = \\frac{1}{|Q|} \\sum_{q\\in Q}\\operatorname{RR}@k(q).\\]\n\nFor example, a first relevant result at position 12 contributes zero to MRR@10 but \\(1/12\\) to MRR@20.\n\nThe denominator normally includes all evaluation queries, including those with no relevant result in the evaluated prefix. Averaging only over successful queries instead measures performance conditional on success and can conceal retrieval failures.\n\nFor fixed ranked lists and relevance judgments, MRR@\\(k\\) cannot decrease as \\(k\\) increases. Nevertheless, scores computed at different cutoffs are not directly interchangeable."
    },
    {
      "heading": "Interpretation and mathematical properties",
      "text": "Strong emphasis on early positions\n\nThe reciprocal discount decreases rapidly near the top of the ranking:\n\nFirst relevant rank | Score\n1 | 1.00\n2 | 0.50\n3 | approximately 0.33\n5 | 0.20\n10 | 0.10\n\nMoving the first relevant result from position \\(r+1\\) to position \\(r\\) increases the query score by\n\n\\[\\frac{1}{r}-\\frac{1}{r+1} = \\frac{1}{r(r+1)}.\\]\n\nThus, moving from second to first place has a much larger effect than moving from tenth to ninth. This expresses a particular preference for early success; it is not, by itself, an empirically established model of user attention.\n\nExpected reciprocal utility\n\nUsing the concept of expected value, MRR can be interpreted as the empirical average of a utility function that gives value \\(1/r\\) to finding the first relevant item at rank \\(r\\), and zero to failure.\n\nThis interpretation does not mean that MRR equals the probability of user satisfaction, the proportion of successful queries, or the expected number of results examined. Those quantities require different definitions or additional behavioral assumptions."
    },
    {
      "heading": "Applications",
      "text": "MRR is commonly used where a system must rank a correct or useful candidate highly.\n\n- Search and question answering: A search engine or question-answering system ranks documents, passages, or candidate answers, and evaluation identifies the first acceptable result.\n- Recommendation: A recommender ranks items for a user or context, often treating a held-out interaction as the target item.\n- Knowledge graph completion: A system ranks possible entities for a missing component of a relation in a knowledge graph.\n- Retrieval-augmented generation: In retrieval-augmented generation, MRR can assess how early a supporting passage appears. It does not directly measure whether the generated answer is correct or adequately supported.\n- Ranking model development: MRR can compare systems trained using learning-to-rank methods or different retrieval and reranking architectures.\n\nIn machine learning, MRR is generally an evaluation metric rather than a directly differentiable loss function. Rank positions change discretely when candidate scores exchange order, so training commonly uses surrogate objectives."
    },
    {
      "heading": "Relationship to other ranking metrics",
      "text": "Hit rate and success at a cutoff\n\nHit rate at \\(k\\), also called success@\\(k\\) in some retrieval settings, records whether at least one relevant result appears among the first \\(k\\) positions. It gives the same credit to success at rank 1 and rank \\(k\\), whereas MRR@\\(k\\) distinguishes them.\n\nFor \\(k\\geq1\\),\n\n\\[\\frac{\\operatorname{HitRate}@k}{k} \\leq \\operatorname{MRR}@k \\leq \\operatorname{HitRate}@k.\\]\n\nAt \\(k=1\\), MRR@1 and hit rate@1 are identical.\n\nAverage precision\n\nAverage precision considers relevant results throughout the ranking, whereas reciprocal rank considers only the first one. Mean average precision, the mean of query-level average precision, is therefore more informative when retrieving multiple relevant items is important.\n\nWhen each query has exactly one relevant item, average precision equals reciprocal rank, provided the evaluation conventions and truncation treatment are the same.\n\nNormalized discounted cumulative gain\n\nNormalized discounted cumulative gain (NDCG) can incorporate graded relevance and rewards relevant items at multiple positions. Standard MRR uses a binary distinction between relevant and nonrelevant results and ignores all relevant results after the first.\n\nMean rank\n\nMean rank averages rank positions directly and can be strongly affected by very poor rankings. MRR bounds every query’s contribution between 0 and 1 and emphasizes top-ranked success. The two metrics can favor different systems because they encode different preferences."
    },
    {
      "heading": "Evaluation choices and limitations",
      "text": "Relevance judgments\n\nMRR depends on what counts as relevant. A passage containing matching words is not necessarily an acceptable answer, and an item without an observed interaction is not necessarily irrelevant.\n\nIncomplete judgments can distort scores if unjudged results are treated as nonrelevant. Queries with no relevant item in the collection also need an explicit policy: including them contributes zeros, while excluding them changes the population being evaluated.\n\nCandidate sets and negative sampling\n\nRanking a target against a small sampled set of alternatives is a different task from ranking it against an entire collection. Sampled evaluation may produce substantially different scores, particularly when difficult alternatives are omitted. Comparisons require compatible candidate pools and sampling procedures.\n\nIn knowledge graph completion, filtered evaluation commonly removes other known true candidates before measuring the target’s rank. Raw and filtered MRR therefore answer different evaluation questions.\n\nTies\n\nEqual candidate scores require a stated tie-handling rule. Deterministic ordering, randomized ordering, and average-rank conventions can yield different reciprocal ranks.\n\nThe reciprocal of an average rank is not generally the same as the expected reciprocal rank under random tie-breaking. For a single relevant item tied across positions 1 and 2, these quantities are \\(2/3\\) and \\(3/4\\), respectively.\n\nLimited coverage of ranking quality\n\nTwo systems can have identical MRR even if one retrieves many additional relevant results and the other retrieves none after its first success. MRR does not measure result diversity, relevance beyond the first hit, response latency, or downstream answer quality.\n\nIt is therefore most informative when early discovery of one relevant result closely matches the task objective.\n\nStatistical and experimental comparability\n\nMRR is an average over an evaluation sample, so a small score difference need not indicate a reliable difference in performance. Query-level paired comparisons and confidence intervals can help characterize uncertainty.\n\nThe evaluation test set should remain separate from model selection and tuning. Data leakage can inflate reported performance, while distribution shift can make a measured MRR unrepresentative of deployment conditions. Meaningful reporting specifies the cutoff, relevance definition, candidate pool, tie policy, and treatment of failed or unanswerable cases."
    }
  ],
  "links": [
    "information-retrieval",
    "recommender-system",
    "expected-value",
    "search-engine",
    "knowledge-graph",
    "retrieval-augmented-generation",
    "learning-to-rank",
    "machine-learning",
    "loss-function",
    "average-precision",
    "normalized-discounted-cumulative-gain",
    "confidence-interval",
    "test-set",
    "data-leakage",
    "distribution-shift"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}