{
  "id": "apache-hadoop",
  "title": "Apache Hadoop",
  "lang": "en",
  "source_lang": "en",
  "summary": "Apache Hadoop is an open-source framework for distributed storage and large-scale data processing across computer clusters.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "product",
  "aliases": [
    "Hadoop"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/apache-hadoop/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/apache-hadoop/",
      "json": "https://aiwiki.page/zh/apache-hadoop.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Apache Hadoop is an open-source software framework for storing and processing large datasets across clusters of networked computers. Developed under the Apache Software Foundation, it combines distributed storage, resource management, and parallel execution facilities. Its core components are Hadoop Common, the Hadoop Distributed File System (HDFS), Yet Another Resource Negotiator (YARN), and an implementation of MapReduce. Together, these components allow workloads to use many machines while accommodating hardware failures as part of normal operation.\n\nHadoop became an important foundation of big data infrastructure because it made large-scale storage and computation practical on clusters of relatively inexpensive servers. Rather than requiring every machine to be exceptionally reliable, it uses replication, monitoring, retries, and coordinated scheduling to maintain service despite individual failures. Its design emphasizes high aggregate throughput and the processing of large datasets, rather than the low latency of individual requests.\n\nThe name also commonly refers to a wider ecosystem of independently developed tools for querying, transforming, coordinating, and managing distributed data. This broader usage can obscure an important distinction: Hadoop itself is neither a single database nor a complete analytics platform. Its significance lies in the infrastructure it provides and in the architectural ideas it helped establish, including moving computation toward stored data, separating resource allocation from application logic, and treating distributed failures as routine."
    },
    {
      "heading": "Origins and development",
      "text": "Hadoop originated in work on Apache Nutch, an open-source search engine project associated with Doug Cutting and Mike Cafarella. A search engine must collect documents, build indexes, and repeatedly process a growing corpus. These requirements exposed the limits of designs that depended on a single machine or tightly constrained storage. Nutch’s developers therefore investigated ways to distribute both files and computation across multiple servers.\n\nTwo papers from Google provided influential architectural models. “The Google File System,” published in 2003, described a distributed storage system designed for large files and unreliable commodity hardware. “MapReduce: Simplified Data Processing on Large Clusters,” published in 2004 by Jeffrey Dean and Sanjay Ghemawat, described a programming model and execution system for distributed data processing. Hadoop implemented related ideas in open-source software; it was not a release of Google’s internal systems or source code.\n\nThe distributed storage and processing components were separated from Nutch in 2006 and developed as Hadoop. Cutting named the project after his young son’s toy elephant, which also explains the elephant imagery associated with it. Yahoo became a major early contributor and deployment site, supporting work on scaling, scheduling, reliability, and production operations. Hadoop became an Apache top-level project in 2008. Its availability under the Apache License 2.0 allowed organizations to use, modify, and redistribute it, including within commercial products and services.\n\nThe project’s major release families reflect changes in architecture rather than merely increases in capacity. Hadoop 1.x, including the 1.0.0 release in 2011, retained the original arrangement in which a JobTracker managed MapReduce execution. Hadoop 2.x, whose 2.2.0 release in 2013 was the first generally available release of that line, established YARN as a general resource-management layer and incorporated major HDFS availability and scaling improvements. Hadoop 3.0.0, released in 2017, introduced further capabilities, including HDFS erasure coding and support for more flexible resource configurations.\n\nCommercial distributions and hosted services subsequently packaged Hadoop with surrounding tools, installation systems, and support. These products helped organizations operate complex clusters, but their contents were not identical to the Apache Hadoop project. Consequently, statements about a “Hadoop platform” can refer to different combinations of Apache components, vendor extensions, and external services. The underlying project remained a collaborative software effort rather than a single vendor’s proprietary platform."
    },
    {
      "heading": "Core architecture and execution environment",
      "text": "Hadoop’s architecture separates three broad concerns: storing data, allocating computing resources, and executing application work. HDFS addresses storage; YARN allocates resources and coordinates application execution; MapReduce supplies one processing model. Hadoop Common provides shared libraries, utilities, configuration mechanisms, and interfaces used across these subsystems. The separation allows a cluster to support different workloads without requiring every application to adopt the same execution engine.\n\nMost core Hadoop services are implemented in Java and run as long-lived processes. Production clusters have commonly used Linux, although Hadoop also supports other environments. A deployment typically distinguishes machines running coordination services from worker machines that hold data and execute tasks. In a traditional combined installation, a worker may run both an HDFS DataNode and a YARN NodeManager. These services cooperate indirectly through clients, schedulers, and knowledge of where data blocks reside.\n\nClients interact with Hadoop through several interfaces. Administrators and users can issue commands through a command-line interface, while programs use filesystem and job-submission APIs. Hadoop also provides web interfaces and HTTP-based access mechanisms for particular services. Configuration is commonly expressed in XML files, including settings for service addresses, storage paths, security, replication, scheduling, and resource limits. Configuration consistency matters because clients and servers must agree on service identities and protocol expectations.\n\nA central design principle is data locality. If a dataset occupies disks across hundreds of machines, moving all its bytes to a separate processing server may consume substantial network bandwidth. Hadoop can instead schedule computation on the machine that stores a required block, or on a nearby machine in the same rack. This is not an absolute requirement: remote reads remain possible, and processing may be scheduled elsewhere when resources are unavailable. Locality is a preference balanced against utilization, fairness, and application progress.\n\nHadoop also provides a filesystem abstraction rather than assuming that every path refers to HDFS. URI schemes and corresponding implementations allow applications to access local filesystems and external storage services through related interfaces. This abstraction supports portability, but it does not make all storage systems behaviorally identical. Rename operations, consistency guarantees, authentication, and access costs may differ. An application that works efficiently on HDFS can therefore behave quite differently when pointed at remote object storage, even if its source code changes little."
    },
    {
      "heading": "HDFS: namespace, blocks, and data access",
      "text": "The Hadoop Distributed File System is a distributed filesystem designed primarily for large files, streaming access, and high throughput. It presents a hierarchical namespace of directories and files, but internally divides file contents into blocks distributed among storage servers. A commonly used default block size in modern Hadoop configurations is 128 MiB, although block sizes are configurable. These blocks are much larger than ordinary filesystem sectors because HDFS is optimized for substantial sequential transfers rather than small random operations.\n\nAn HDFS deployment separates namespace management from block storage. The NameNode maintains information about directories, filenames, permissions, file-to-block mappings, and other metadata. DataNodes store the blocks themselves and serve read and write requests. A client first obtains location information from the NameNode and then transfers file data directly to or from DataNodes. The NameNode therefore coordinates access without becoming the normal transit point for the full contents of every file.\n\nFor replicated files, a write generally proceeds through a pipeline. The client sends data to one DataNode, which forwards it to another, continuing through the selected replica chain. Acknowledgments return through that pipeline. A replication factor of three is a common default, meaning that each block has three stored copies, although individual files can use different factors. Placement policies account for network topology so that replicas are not unnecessarily concentrated in a single failure domain, such as one rack.\n\nReading follows a different path. After obtaining block locations, the client selects a suitable replica, often preferring a nearby one. It verifies data using checksums and can switch to another replica if a read fails or corruption is detected. HDFS thus combines distributed placement with end-to-end checks that help distinguish valid bytes from damaged storage contents. A large file can be processed concurrently because separate tasks can read different blocks from different DataNodes.\n\nHDFS does not reproduce every behavior of a local POSIX filesystem. Its model favors a single active writer for a file, sequential output, and subsequent large-scale reading; append is supported, but arbitrary in-place modification is not its central abstraction. Hadoop also distinguishes between pushing buffered data through the pipeline and requesting stronger synchronization through operations such as hflush and hsync. Applications that require low-latency updates to individual records usually place a database or another specialized storage layer above HDFS rather than treating it as a conventional mutable disk."
    },
    {
      "heading": "Storage resilience, availability, and scaling",
      "text": "HDFS assumes that disks, machines, and network connections can fail during ordinary operation. DataNodes send heartbeats and block reports to the NameNode, allowing it to track service health and block placement. If a DataNode becomes unavailable, the NameNode identifies blocks whose replica counts have fallen below their targets and arranges additional replication from surviving copies. This process repairs redundancy asynchronously; it does not make simultaneous loss of all copies recoverable.\n\nReplication addresses several distinct risks but is not equivalent to backup. An accidental deletion, an authorized destructive operation, or a software error may affect all replicas through the same namespace. HDFS snapshots can preserve point-in-time views of directories using metadata-efficient techniques, while copies to another cluster or storage system provide a separate recovery boundary. Cross-cluster copying is often performed with DistCp, a Hadoop utility that parallelizes large copying jobs. These mechanisms have different retention, isolation, and recovery properties.\n\nEarly Hadoop deployments had a single active NameNode whose failure interrupted namespace service. HDFS high availability introduced active and standby NameNodes, with namespace changes shared through mechanisms such as a quorum of JournalNodes. The standby tracks the state needed to take over. Automatic failover commonly uses Apache ZooKeeper for coordination together with failover controllers. Fencing mechanisms prevent an old active process from continuing to act as a writer after another NameNode has assumed control.\n\nThe Secondary NameNode, despite its name, is not a ready-to-use standby NameNode. Its traditional role is checkpointing: it combines a namespace image with accumulated edit records to produce an updated checkpoint. Confusing checkpointing with high availability can leave a deployment without the expected failover protection. HDFS federation addresses another problem by allowing multiple independent namespaces and block pools to share a set of DataNodes. Federation distributes namespace responsibility; high availability protects the service of a particular namespace.\n\nHadoop 3 added erasure coding as a storage option for HDFS. Rather than keeping complete copies, an erasure-coded block group stores data units and computed parity units. For example, a six-data-unit, three-parity-unit policy has a nominal storage overhead of \\(9/6 = 1.5\\), compared with \\(3\\) for three-way replication, and can tolerate up to three missing units in a block group. The saving comes with trade-offs: encoding, reconstruction, and some access patterns require additional computation and network traffic. Replication therefore remains useful for workloads and files whose behavior does not suit erasure-coded storage."
    },
    {
      "heading": "MapReduce processing and fault recovery",
      "text": "Hadoop MapReduce implements a model of parallel computation organized around key–value pairs. A map function processes input records and emits intermediate pairs. The framework partitions these pairs, transfers them to reducers, sorts them by key, and groups values associated with the same key. A reduce function then processes each group. The programmer specifies the record-level transformations, while the framework handles task distribution, intermediate transfers, and much of the recovery from execution failures.\n\nWord counting illustrates the model. For each encountered word, a mapper emits a pair such as (word, 1). During the shuffle and sort phase, all pairs for a particular word are routed to the same reducer and grouped together. The reducer sums their values to produce the final count. Although simple, this example captures a general pattern used in aggregation, log analysis, inverted-index construction, and some forms of joins. An input format determines how files become logical splits and records; a split need not correspond exactly to an HDFS block.\n\nMap output is typically buffered in memory and spilled to local disk before being merged for transfer. The partitioner decides which reducer receives a key, while the sort and grouping comparators determine ordering and grouping behavior. A combiner can sometimes aggregate mapper output before transfer, reducing network traffic. Its use is constrained: the computation must remain correct whether combining occurs zero, one, or multiple times. For an average, emitting a partial sum and count is safe, whereas repeatedly averaging intermediate averages generally is not.\n\nFault recovery relies substantially on re-execution. If a task attempt fails, another attempt can process the same input. If a machine holding intermediate map output is lost, the relevant map tasks may have to run again because that intermediate output is normally stored locally rather than replicated through HDFS. Speculative execution can launch duplicate attempts for unusually slow tasks, allowing the faster attempt to supply the accepted result. It can help with slow machines, but it does not necessarily solve skew caused by an intrinsically oversized partition.\n\nThese mechanisms favor functions whose effects are reproducible or carefully controlled. Writing to an external database from inside a task can create duplicates when attempts are retried; accepting one output attempt does not automatically provide exactly-once effects in arbitrary external systems. MapReduce also incurs scheduling, serialization, sorting, network, and disk costs. Chaining many jobs materializes intermediate results repeatedly, making the model less efficient for iterative algorithms and interactive exploration than engines designed to retain reusable state or execute richer computation graphs."
    },
    {
      "heading": "YARN and cluster resource management",
      "text": "YARN, short for Yet Another Resource Negotiator, separates resource management from a particular processing model. In the original Hadoop architecture, the JobTracker combined cluster-wide scheduling with the management of MapReduce jobs. This concentrated responsibilities in one service and tied the cluster closely to MapReduce. YARN divides those responsibilities among a ResourceManager, NodeManagers, and per-application ApplicationMasters, allowing different processing frameworks to share the same resource infrastructure.\n\nThe ResourceManager arbitrates cluster resources. A NodeManager runs on each worker and reports available capacity, monitors local execution, and launches assigned work. An ApplicationMaster manages one application’s progress and requests resources from the ResourceManager. In a MapReduce application, the ApplicationMaster coordinates map and reduce attempts; a different engine supplies an ApplicationMaster suited to its own execution model. Failure recovery is consequently split between cluster services and application-specific logic rather than controlled by one universal job coordinator.\n\nYARN allocates resources in containers. Here, “container” primarily means an allocation of resources and an execution context; it is not synonymous with a Docker container or a virtual machine. A container may represent a specified amount of memory and virtual CPU capacity, with additional resource types available under suitable configurations. Process launch, isolation, accounting, and enforcement depend on the configured executor and operating-system facilities. The scheduler’s accounting of a virtual core is also not a guarantee that an application receives a dedicated physical processor at every instant.\n\nScheduling policies determine how organizations share a cluster. Capacity-based scheduling assigns resources through queues with configured capacities, priorities, and limits, while fair-sharing approaches seek to distribute resources among competing applications or groups over time. Hierarchical queues can reflect departments or workloads, and preemption can reclaim resources when one queue’s use prevents another from obtaining its allocation. Resource allocation must also account for locality, application demand, and multiple resource dimensions rather than simply dividing a count of identical slots.\n\nYARN is a resource-management framework, not an automatic solution to every performance or reliability problem. Applications must still respond correctly to container loss, implement their own progress tracking, and avoid requesting resources in ways that prevent useful concurrency. A workload that requests excessive memory may wait despite spare CPU capacity; a workload that underestimates memory may be terminated for exceeding limits. High availability for the ResourceManager and recovery support for application management reduce service disruption, but the precise recovery behavior remains dependent on the processing framework and its configuration."
    },
    {
      "heading": "Data formats and the surrounding ecosystem",
      "text": "Much of Hadoop’s practical value comes from tools built around its storage and execution interfaces. Apache Hive provides a SQL-oriented layer over large datasets and maintains table definitions in a metastore. A table can describe files stored in HDFS or another supported filesystem rather than owning data through the storage architecture of a traditional relational database. Hive historically compiled queries into MapReduce jobs and later supported other execution engines. Its existence does not mean that HDFS itself understands SQL.\n\nApache HBase addresses a different access pattern. Inspired by Google’s Bigtable design, it provides a distributed, column-family-oriented database commonly backed by HDFS. It supports keyed access and updates that are unsuitable for direct implementation as whole-file batch transformations. HBase uses additional services and mechanisms, including write-ahead logging, memory-resident structures, and compaction of persistent files. Consequently, its latency, consistency, and operational behavior must be understood as database properties, not inferred directly from those of HDFS.\n\nFile formats strongly influence processing efficiency. Apache Avro supports schema-based serialization and is frequently used for record-oriented exchange and storage. Apache Parquet and Apache ORC are columnar formats designed to let engines read selected columns and skip irrelevant portions of data. Their encodings and statistics can reduce both I/O and decoding work. A query that selects three columns from a hundred-column dataset may avoid reading much of the stored content, although actual savings depend on layout, predicates, compression, and engine support.\n\nCompression and partitioning interact with parallel execution. A compressed stream that cannot be split at arbitrary positions may limit how many tasks can independently process a single file. Columnar formats address splitting through their internal structures, such as row groups or stripes. Directory partitioning can also restrict scans: files organized by date may allow an engine to ignore unrelated days. These benefits require sensible partition granularity, because excessive partitioning can produce many tiny files and heavy metadata-management costs.\n\nOther ecosystem projects provide coordination, workflow execution, ingestion, and administration. Their boundaries matter: ZooKeeper is a separate coordination service, not a Hadoop storage daemon, and Apache Spark is a separate processing engine that can run on YARN and access HDFS without being part of Hadoop’s core. A “Hadoop ecosystem” installation therefore consists of interoperating projects rather than a single uniform product. Compatibility depends on interfaces, library versions, serialization formats, and service configuration, while features such as table-level transactions are supplied by particular engines or table layers rather than by an ordinary directory of files alone."
    },
    {
      "heading": "Security, administration, and operational control",
      "text": "A Hadoop cluster combines storage services, resource schedulers, execution workers, clients, and administrative interfaces, creating several distinct security boundaries. In a simple, non-secure configuration, Hadoop can rely on user identities supplied by clients rather than strongly authenticating them. Such a setup must not be confused with a deployment designed to resist hostile users. Secure operation requires coordinated identity, authorization, transport protection, and service configuration rather than the activation of one universal security switch.\n\nKerberos is the principal authentication mechanism in traditional secure Hadoop deployments. Users and services obtain credentials that allow them to establish authenticated identities. Delegation tokens then permit applications and services to access Hadoop resources without repeatedly presenting a user’s long-term credentials during distributed execution. Service principals and keytab files support daemon authentication, while token lifetimes and renewal affect the behavior of long-running jobs. Authentication establishes identity; it does not by itself determine which files or queues that identity may access.\n\nAuthorization operates at multiple layers. HDFS provides ownership, group membership, permission bits, and access-control lists. YARN controls access to queues and application-management operations. Higher-level systems may add permissions for databases, tables, columns, or other logical objects. Administrative interfaces, proxy-user rules, and service accounts require separate attention because a broadly privileged service can become a route around narrower user-level restrictions. Centralized policy tools can coordinate some of these controls, but the enforcement points remain distributed across services.\n\nProtection of data in transit and at rest also involves separate mechanisms. Hadoop supports encrypted communication for relevant service paths and can use TLS for web endpoints. HDFS encryption zones provide transparent encryption of file contents using keys managed through a key-management service. Such encryption does not eliminate exposure through authorized reads, compromised clients, or all forms of metadata. Similarly, disk redundancy protects availability but does not provide confidentiality or prevent an authenticated user from reading permitted content.\n\nOperational control depends on visibility into both distributed services and workloads. Administrators examine metrics, logs, audit records, block health, queue occupancy, garbage-collection behavior, and disk capacity. Decommissioning allows a DataNode’s blocks to be relocated before the machine is removed. Rack topology configuration influences replica placement, and balancing tools redistribute data after hardware changes. Rolling upgrades and rollback facilities reduce disruption for supported transitions, but version compatibility and persisted metadata formats still constrain change. These tasks make cluster administration a continuing systems-engineering activity rather than a one-time installation procedure."
    },
    {
      "heading": "Workloads, performance, and practical limits",
      "text": "Hadoop is especially suited to workloads that process substantial portions of large datasets and can tolerate execution measured in seconds, minutes, or hours rather than milliseconds. Examples include transforming event logs, building search indexes, reconciling large collections of records, preparing training datasets, and producing periodic aggregates for a data warehouse. Its strengths are most apparent when work can be partitioned across many machines and when the value of aggregate throughput exceeds the need for immediate individual responses.\n\nPerformance depends on the complete path from storage through execution to output. A workload may be limited by disk throughput, network bandwidth, decompression, serialization, CPU processing, or scheduler delays. Adding workers helps only when the limiting stage can use them. In simplified form, elapsed time can be represented as\n\n\\[T \\approx T_{\\text{startup}} + T_{\\text{read}} + T_{\\text{compute}} + T_{\\text{shuffle}} + T_{\\text{write}},\\]\n\nalthough actual stages often overlap. This decomposition explains why a small input can be dominated by launch overhead while a large aggregation is dominated by shuffle traffic.\n\nData skew is a recurring source of imbalance. If one key accounts for a large fraction of intermediate records, the reducer receiving that key may run far longer than its peers. Increasing the number of reducers does not automatically split that single key’s group. Remedies may require changing the algorithm, introducing partial aggregation, or deliberately dividing a heavy key into subgroups followed by another aggregation. Similar imbalance arises when input partitions differ greatly in size or computational difficulty.\n\nThe small-file problem follows from both metadata and execution overhead. Each HDFS file and block consumes namespace-management resources, even when the file contains little data. A large population of tiny files can therefore exhaust NameNode memory or create expensive listing and opening operations before disk capacity is exhausted. It can also generate many short tasks whose scheduling costs outweigh useful computation. Consolidated files and appropriate storage formats reduce these costs, but the suitable file size depends on the engine, update pattern, and desired parallelism.\n\nHadoop is not a general replacement for every database or computing system. Traditional transaction processing needs features such as indexed point lookups, fine-grained updates, and transaction semantics that basic HDFS and MapReduce do not provide. Iterative machine-learning algorithms may benefit from engines that retain data in memory, while accelerator-heavy model training has additional communication and resource-management requirements. Hadoop remains useful within such pipelines for durable storage and large-scale preparation, but its suitability must be evaluated at the level of the specific subsystem and workload rather than the broad label “big data.”"
    },
    {
      "heading": "Cloud deployment and changing architectural roles",
      "text": "Traditional Hadoop deployments colocated storage and computation on the same worker machines. This arrangement made data locality valuable and allowed local disks to supply large aggregate bandwidth. However, storage and compute capacity then tended to scale together: an organization needing more storage might add machines whose processing capacity was not required, while a temporary computation surge could demand workers whose disks would be underused. Hardware procurement, capacity planning, and cluster maintenance were closely coupled.\n\nCloud computing made a different arrangement widely practical: durable data can reside in object storage, while processing clusters are created, expanded, or removed separately. Hadoop filesystem connectors expose these stores through interfaces familiar to Hadoop applications. This supports transient compute clusters and independent storage growth, but it weakens the original assumption that a task can run beside a local replica. Performance instead depends on network access, remote request behavior, connector implementations, caching, and the characteristics of the object store.\n\nFilesystem compatibility in this setting is functional rather than exact. On HDFS, renaming a path is a namespace operation; in many object-storage arrangements, a rename-like operation requires copying objects and deleting old ones. Output-commit protocols designed around inexpensive atomic renames may consequently be slow or inappropriate. Object-store-aware committers and related protocols address this mismatch. The lesson is not that remote storage is inherently inferior, but that correctness and efficiency depend on matching application assumptions to storage semantics.\n\nThe wider data lake approach also evolved beyond the original Hadoop stack. Organizations can retain large collections of structured and unstructured data in shared storage while using multiple engines for SQL, batch processing, streaming, and learning workloads. Catalogs and table-management layers supply schema, transaction, versioning, and maintenance capabilities above the files. Hadoop contributed important interfaces and operational experience to this architecture, but neither a data lake nor a modern analytical table necessarily requires HDFS, YARN, or MapReduce.\n\nHadoop’s components therefore have distinct deployment trajectories. A system may use HDFS with a non-MapReduce engine, run Spark on YARN, or use Hadoop client libraries solely to access external storage. Managed services may expose selected Hadoop capabilities without requiring customers to administer the underlying machines. Conversely, some platforms replace the entire core stack while retaining formats, conventions, or software descended from its ecosystem. Precise descriptions distinguish these cases, because using Hadoop-compatible storage interfaces is not the same as operating a Hadoop cluster, and replacing MapReduce does not by itself imply that HDFS or YARN has also been removed."
    }
  ],
  "links": [
    "open-source-software",
    "mapreduce",
    "big-data",
    "database",
    "search-engine",
    "google",
    "linux",
    "application-programming-interface",
    "file-system",
    "metadata",
    "data-backup",
    "parallel-computing",
    "virtual-machine",
    "operating-system",
    "sql",
    "apache-parquet",
    "apache-spark",
    "kerberos",
    "authentication",
    "transport-layer-security",
    "data-warehouse",
    "algorithm",
    "machine-learning",
    "cloud-computing",
    "data-lake"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}