{
  "id": "mapreduce",
  "title": "MapReduce",
  "lang": "en",
  "source_lang": "en",
  "summary": "MapReduce is a programming model and execution framework for processing large datasets through parallel mapping, key-based grouping, and reduction across distributed computers.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "MapReduce programming model",
    "Map/Reduce"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/mapreduce/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/mapreduce/",
      "json": "https://aiwiki.page/zh/mapreduce.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "MapReduce is a programming model for processing large datasets as collections of key–value pairs, together with an execution approach that distributes the work across multiple computers. A programmer supplies a map function that produces intermediate key–value pairs and a reduce function that processes the values associated with each intermediate key. The execution framework handles task scheduling, data transfer, grouping, and recovery from many worker failures. MapReduce became an influential approach to distributed computing and big data processing after Google described its implementation in 2004."
    },
    {
      "heading": "Programming model",
      "text": "A MapReduce computation separates application logic from much of the machinery needed for parallel execution. Rather than explicitly coordinating communication between workers, the programmer describes how individual records are transformed and how records sharing a key are combined.\n\nThe core functions can be expressed as:\n\n\\[\\operatorname{map}(k_1,v_1) \\rightarrow [(k_2,v_2)]\\]\n\n\\[\\operatorname{reduce}(k_2,[v_2]) \\rightarrow [(k_3,v_3)]\\]\n\nThe brackets denote a collection of values or pairs, not a requirement that all values be materialized in memory. Many implementations expose reducer inputs through an iterator.\n\nThe key and value types can differ between the input, intermediate, and output stages. A map invocation may emit no pairs, one pair, or many pairs. A reduce invocation may likewise produce zero or more output records.\n\nBetween these functions, the framework performs an essential operation: it gathers all intermediate values belonging to the same key and supplies them to the same reduce invocation. This grouping and movement of data is commonly called the shuffle.\n\nMapReduce is therefore more than an independent map followed by an independent reduction over an entire dataset. Its defining feature is the key-based redistribution between those stages.\n\nWord-count example\n\nA standard example counts word occurrences in a document collection. For each document, the mapper emits a pair consisting of a word and the integer one. The framework groups identical words, and the reducer sums their counts.\n\nmap(document_id, text):\n    for each word in text:\n        emit(word, 1)\n\nreduce(word, counts):\n    emit(word, sum(counts))\n\nFor the input:\n\nred blue red\nblue green\n\nthe intermediate groups are conceptually:\n\nblue  → [1, 1]\ngreen → [1]\nred   → [1, 1]\n\nThe output contains blue → 2, green → 1, and red → 2. The order of these groups is not fundamental to the model.\n\nPractical word counting also depends on how words are identified, including decisions about punctuation, case, and language. Those rules belong to the application rather than to MapReduce itself."
    },
    {
      "heading": "Execution pipeline",
      "text": "A MapReduce job usually passes through several stages. The exact implementation varies between systems, but the following structure is common.\n\nInput splitting and mapping\n\nThe input is divided into logical splits, and each split becomes input to a map task. An input reader converts the split into records suitable for the map function.\n\nA split need not correspond exactly to a physical storage block or an entire file. Record readers may handle boundaries so that a logical record, such as a line of text, is not incorrectly divided between map invocations.\n\nMap tasks can run independently. A scheduler assigns them to available workers and may place them near their input data.\n\nPartitioning, shuffle, and sorting\n\nEach intermediate key is assigned to a reducer partition. A common partitioning rule uses a hash function:\n\n\\[p(k)=h(k)\\bmod R,\\]\n\nwhere \\(R\\) is the number of reduce partitions.\n\nAll keys that compare equal must be assigned to the same partition. Otherwise, their values could be processed by different reducers, breaking the expected grouping semantics.\n\nMapper output is usually buffered and may be sorted and spilled to local storage. Reducers fetch the partitions assigned to them from the map workers. These transfers constitute much of the shuffle.\n\nImplementations commonly sort intermediate records by key and merge the fetched segments before reduction. Sorting is an implementation technique for grouping, and can also support applications that require ordered keys. However, a collection of separately sorted reducer outputs is not automatically one globally sorted sequence.\n\nReduction and output\n\nEach reducer processes its assigned keys and associated values, then writes output through an output-format mechanism.\n\nA job may have many map tasks and many reduce tasks, even when the cluster has fewer execution slots. Workers execute tasks in successive waves. Some frameworks also support map-only jobs, which omit the shuffle and reduction stages when each input record can be processed independently.\n\nThe shuffle can overlap with the execution of map tasks, but reduction for a complete key group must account for all relevant map output. MapReduce should therefore not be understood as necessarily running every stage in strictly non-overlapping intervals."
    },
    {
      "heading": "Combiners and aggregation",
      "text": "A combiner performs optional local aggregation of mapper output before it is transferred across the network. In word counting, a mapper can replace many pairs such as (word, 1) with a single partial count.\n\nA correct combiner must preserve the final result despite variations in how often it is invoked and how records are grouped. In common implementations, it may run zero times, once, or multiple times. An application must not depend on its execution.\n\nAssociative and commutative operations, such as integer addition, are natural candidates for local combination. More generally, a computation can use a mergeable intermediate representation.\n\nFor example, averaging local averages is generally incorrect when the groups contain different numbers of observations. Instead, each group can produce a pair consisting of its sum and count:\n\n\\[(s_1,n_1)\\oplus(s_2,n_2) = (s_1+s_2,n_1+n_2).\\]\n\nThe final average is:\n\n\\[\\bar{x}=\\frac{s}{n}.\\]\n\nThe reduce function itself need not always be associative or commutative. These properties become important when partial results are combined or reordered. Floating-point addition also requires care: finite-precision rounding means that different aggregation orders can yield slightly different numerical results."
    },
    {
      "heading": "Distribution and fault tolerance",
      "text": "MapReduce frameworks are designed to execute computations on clusters where individual machines or processes may fail during a job.\n\nData locality\n\nEarly large-scale implementations emphasized data locality: scheduling computation on or near machines storing the required input. This reduces network traffic by moving code to data rather than transferring every input record to an arbitrary worker.\n\nA distributed file system can support this approach by storing replicated blocks across machines and exposing their locations to the scheduler. Locality depends on the deployment architecture; systems using remote storage may place greater emphasis on network bandwidth and disaggregated resource management.\n\nTask re-execution\n\nA common recovery strategy is to rerun failed tasks. Input data remains available in durable storage, while intermediate data may be reconstructed by repeating the map tasks that generated it.\n\nIn the original Google design, map outputs were stored on worker-local disks, whereas completed reduce outputs were written to the distributed file system. Losing a map worker could therefore require regenerating its intermediate output even if the corresponding map task had previously completed.\n\nDeterministic task functions make such recovery easier to reason about. External side effects require additional safeguards: retrying a task that sends a message or modifies an external database can repeat that action. A framework’s output-commit mechanism does not automatically make arbitrary external operations execute exactly once.\n\nSlow tasks and speculative execution\n\nA small number of unusually slow tasks, often called stragglers, can delay an otherwise nearly finished job. Some implementations launch additional attempts of slow tasks on other workers and accept a successful attempt.\n\nThis technique, known as speculative execution, can shorten completion time when slowness is caused by a particular machine. It is less effective when a task is slow because its partition genuinely contains much more work than others.\n\nWorker recovery does not by itself eliminate all single points of failure. Availability also depends on the design of coordinators, storage services, and job metadata management."
    },
    {
      "heading": "History and implementations",
      "text": "The names map and reduce draw on operations familiar from functional programming. MapReduce combined related ideas with distributed execution, key-based grouping, automatic scheduling, and failure recovery at cluster scale.\n\nJeffrey Dean and Sanjay Ghemawat described Google’s system in the 2004 paper MapReduce: Simplified Data Processing on Large Clusters, presented at the Symposium on Operating Systems Design and Implementation. The paper documented a framework used for large-scale data processing at Google, including tasks associated with web indexing.\n\nApache Hadoop subsequently made MapReduce widely available as open-source software. Hadoop combined a MapReduce execution framework with the Hadoop Distributed File System and an expanding ecosystem of data-processing tools.\n\nEarly Hadoop used a JobTracker and TaskTrackers to coordinate execution. Hadoop’s later YARN architecture separated cluster resource management from application-specific coordination. Hadoop MapReduce could then operate as an application framework on shared cluster infrastructure.\n\nMapReduce is not synonymous with Hadoop. It denotes a programming model and a family of execution techniques; Hadoop MapReduce is a particular implementation. Likewise, Hadoop’s storage and resource-management components can support processing engines other than MapReduce."
    },
    {
      "heading": "Applications",
      "text": "MapReduce is particularly well suited to large, bounded datasets for which substantial work can be performed independently before key-based aggregation. Common applications include:\n\n- Log processing: computing request counts, traffic summaries, and grouped event statistics.\n- Search indexing: building an inverted index that maps terms to documents, an important structure in information retrieval.\n- Data preparation: filtering, transforming, deduplicating, and reorganizing records for data engineering workflows.\n- Relational processing: implementing grouping, aggregation, and some join strategies.\n- Large-scale analysis: calculating histograms, frequency distributions, and other descriptive statistics.\n- Machine-learning preparation: constructing features and processing training data.\n\nA join can be implemented by emitting records from two datasets under the same join key, tagging their origin, and matching them in a reducer. Alternative strategies can avoid a full shuffle when a dataset is small enough to distribute to workers or when inputs already have compatible partitioning.\n\nSome machine-learning and graph algorithms can be expressed as sequences of MapReduce jobs. Whether this is efficient depends on the amount of communication, the size of intermediate results, and the number of repeated passes."
    },
    {
      "heading": "Performance and limitations",
      "text": "Communication and storage costs\n\nMapping may scale well while the shuffle becomes a bottleneck. Large intermediate datasets consume network bandwidth, sorting resources, and local storage.\n\nPerformance depends not only on processor speed or asymptotic computational complexity, but also on data volume, serialization, partitioning, compression, and storage throughput. Increasing the number of workers does not necessarily produce a proportional speedup.\n\nData skew\n\nA frequently occurring key can direct a disproportionate amount of work to one reducer. Because all values for that key normally belong to one logical group, adding reducers does not automatically solve the imbalance.\n\nApplications sometimes split a heavy key into temporary subkeys and merge partial results in another stage. This approach works only when the computation permits that decomposition.\n\nIterative workloads\n\nTraditional MapReduce systems commonly materialize job outputs to durable storage between jobs. Repeatedly writing and rereading the same data can be expensive for iterative optimization, graph traversal, and interactive analysis.\n\nApache Spark and other dataflow engines support more flexible execution graphs and can retain reusable data in memory. They address workloads that do not fit efficiently into repeated map–shuffle–reduce jobs, though they still face communication costs and memory limits.\n\nLatency and streaming\n\nMapReduce is principally associated with batch processing. Task startup, scheduling, shuffle, and output management introduce overheads that make conventional implementations less suitable for small interactive requests or continuously updated results.\n\nStream-processing systems add concepts such as event time, windows, and continuously maintained state. These are not part of the basic MapReduce model.\n\nExpressiveness and optimization\n\nMapReduce can express a broad range of computations through multiple jobs, but expressibility does not imply convenience or efficiency. Complex workflows may require substantial intermediate data and manual coordination.\n\nIn a relational database, a declarative SQL query allows an optimizer to choose among alternative execution plans using knowledge of relational operations. Arbitrary map and reduce functions can conceal application semantics from the framework, limiting some forms of automatic optimization. Higher-level systems built over distributed processing engines recover part of this information by representing operations explicitly.\n\nMapReduce remains useful as an abstraction for understanding partitioned computation, grouped aggregation, and recovery through re-execution. Its practical suitability depends on whether a workload benefits from those properties more than it is constrained by shuffle costs, job boundaries, and repeated data materialization."
    }
  ],
  "links": [
    "distributed-computing",
    "big-data",
    "google",
    "parallel-computing",
    "hash-function",
    "database",
    "functional-programming",
    "apache-hadoop",
    "open-source-software",
    "information-retrieval",
    "data-engineering",
    "statistics",
    "training-data",
    "machine-learning",
    "algorithm",
    "computational-complexity",
    "apache-spark",
    "relational-database",
    "sql"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}