{
  "id": "apache-iceberg",
  "title": "Apache Iceberg",
  "lang": "en",
  "source_lang": "en",
  "summary": "Apache Iceberg is an open table format that supports transactional, versioned analytical datasets across distributed storage systems and processing engines.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "product",
  "aliases": [
    "Iceberg",
    "Apache Iceberg table format"
  ],
  "generated_at": "2026-11-06",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/apache-iceberg/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/apache-iceberg/",
      "json": "https://aiwiki.page/zh/apache-iceberg.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Apache Iceberg is an open-source table format for large analytical datasets. It specifies how data files, schemas, partitions, and table versions are described and coordinated so that independent processing engines can read and modify the same logical table. Unlike a file format such as Apache Parquet, which defines the representation of records within an individual file, Iceberg defines the organization and evolution of a table that may contain millions of files. It is neither a standalone database management system nor a query engine.\n\nIceberg brings features associated with analytical databases—including atomic table updates, schema evolution, consistent snapshots, and historical queries—to data stored in distributed file systems and cloud object stores. These capabilities make it an important component of data lakes and data lakehouses, where storage and computation are managed separately. Its design replaces assumptions about directory names and file listings with explicit, versioned metadata, allowing engines to identify precisely which files belong to a table at a particular point in time.\n\nThe project includes a public format specification, software libraries, and integrations with processing systems. Its principal technical concerns are table correctness under concurrent access, scalable query planning, evolution without unnecessary data rewriting, and interoperability across implementations. The guarantees of a deployed Iceberg table depend not only on the format, but also on the catalog, storage system, execution engine, and operational policies surrounding it."
    },
    {
      "heading": "Origins and architectural purpose",
      "text": "Iceberg originated at Netflix in work led by Ryan Blue and Daniel Weeks to address limitations of large tables managed through Apache Hive conventions. Work began in 2017, and the project entered the Apache Software Foundation’s incubator in November 2018. It became an Apache top-level project in May 2020. It is distributed under the Apache License 2.0 and developed through the foundation’s community governance model.\n\nTraditional Hive-style tables commonly identify partitions through directory paths, with a metastore recording partition locations. A table partitioned by date might store files beneath paths resembling day=2025-01-01/. Query engines discover the relevant files through combinations of metastore operations and directory listings. At very large scale, enumerating partitions and listing files can become expensive. More importantly, directory contents alone do not provide a transactional definition of which files should be visible together.\n\nThis arrangement creates correctness problems when a writer replaces multiple files. If readers independently enumerate storage while the replacement is underway, they may encounter an incomplete or inconsistent combination of old and new files. Storage-specific rename behavior and historical differences in listing consistency further complicated implementations. Iceberg’s response was not to make directory discovery more elaborate, but to remove it from the definition of table state: a committed metadata version explicitly identifies the files belonging to each snapshot.\n\nThe distinction between a table format and a storage engine is central to this design. Iceberg does not require a dedicated database server to own every data read. Files can remain in an object store or distributed file system, while engines such as Apache Spark, Trino, and Apache Flink plan and execute queries. Libraries and catalogs coordinate metadata changes rather than routing all data through a single service.\n\nThis architecture connects Iceberg to the separation of storage and compute in cloud computing. Different compute clusters can access a shared table without sharing local disks or an execution runtime. The resulting flexibility does not eliminate coordination: it relocates coordination to table metadata, commit protocols, and catalog services. Iceberg therefore occupies a layer between physical data files and the systems that interpret them as an evolving analytical table."
    },
    {
      "heading": "Metadata hierarchy and table representation",
      "text": "An Iceberg table is represented by a hierarchy of metadata rather than by the contents of a directory. At the top, a catalog or another supported table-loading mechanism identifies the current table metadata file. This file, encoded in JSON, records information such as the table’s unique identifier, schemas, partition specifications, properties, snapshot references, and the current snapshot identifier. Replacing the authoritative metadata reference publishes a new table state.\n\nA snapshot represents the table’s data state at a particular point in its history. It refers to a manifest list, which identifies manifest files and includes summary information used during planning. A manifest is an Apache Avro file containing entries for data files or delete files. Entries describe file paths, partition values, record counts, file sizes, and column-level metrics, together with information about their addition or removal. Manifests distinguish data-file content from delete-file content.\n\nThis hierarchy allows substantial metadata reuse. An append operation can add manifests describing new files while retaining existing manifests. It does not need to rewrite a complete list of all records or copy all historical table data. Manifest lists also carry partition summaries, allowing a planner to reject some manifests before opening them. The design distributes metadata across manageable files rather than requiring one ever-growing file to describe every detail.\n\nData files commonly use Parquet, although the Iceberg specification also supports Avro and ORC. These formats handle physical encoding, compression, and file-internal structures. Iceberg adds table-level meaning: which schema applies, which partition specification produced a file, whether a file remains live, and which delete information affects its rows. A collection of Parquet files is therefore not automatically an Iceberg table; the governing metadata and commit protocol are essential.\n\nMost files participating in this structure are immutable after creation. Updates produce new metadata and, when necessary, new data or delete files rather than modifying published files in place. This makes historical states reproducible and avoids exposing partially modified files to readers. Immutability also explains why storage reclamation is separate from logical deletion: an obsolete file may still be required by an older snapshot.\n\nThe metadata hierarchy is not a general-purpose row index. It primarily identifies candidate files and their relationships to table versions. File-level statistics can make scans selective, but Iceberg does not inherently provide the point-lookup behavior of a B-tree or a dedicated database index."
    },
    {
      "heading": "Schema evolution and stable field identity",
      "text": "Iceberg identifies columns by persistent numeric field identifiers rather than relying solely on names or physical positions. This distinction is fundamental to schema evolution. A column renamed from customer_id to account_id retains its identity, so files written before the rename can still be interpreted correctly. Reordering columns similarly changes the logical presentation without changing which stored values belong to which fields.\n\nName-based interpretation can fail when a dropped column name is later reused. Suppose a table originally contains a string column named status, drops it, and subsequently adds a different column also called status. Iceberg assigns the new field a new identifier. Old values are not silently reassigned to the new field merely because the names match. Field identifiers also apply within nested structures, including struct fields and components of collection types.\n\nMany schema changes are metadata operations rather than full-table rewrites. Adding an optional column does not require inserting physical values into every existing file. Readers project the current schema onto older files, interpreting absent fields according to the applicable schema and default-value rules. Dropping a column removes it from the current logical schema without immediately erasing its physical bytes from existing files.\n\nType evolution is deliberately constrained. Supported promotions include widening an integer to a long integer, a float to a double, and increasing decimal precision while keeping the scale unchanged. Arbitrary conversions, such as treating an existing string column as a timestamp, are not equivalent metadata-only changes. Such transformations require explicit processing because they may alter meaning, fail for some values, or lose information.\n\nFor files imported from systems that did not write Iceberg field identifiers, name mapping can provide a bridge between file field names and Iceberg field IDs. This is especially relevant when registering existing datasets rather than rewriting them immediately. Correct interpretation still depends on the mapping accurately reflecting the historical files; stable identifiers cannot retrospectively resolve every ambiguity in inconsistent source data.\n\nSchema identity also differs from business identity. Iceberg can record identifier fields intended to identify rows, but this does not make the format a relational constraint-enforcement system. An identifier-field declaration is not, by itself, an enforced primary key, and it does not automatically prevent duplicate records. Uniqueness, referential integrity, and application-level validation generally remain responsibilities of writers, processing systems, or surrounding data-management services."
    },
    {
      "heading": "Hidden partitioning and layout evolution",
      "text": "Iceberg separates logical query columns from physical partition values through hidden partitioning. A table can be partitioned using a transformation such as day(event_time) while exposing event_time as an ordinary timestamp column. A query filtering that timestamp does not need to include a separate, manually maintained date-partition column. The planner derives a partition filter from the original predicate whenever the transformation permits it.\n\nPartition transformations include identity, year, month, day, hour, bucket, and truncate, subject to type compatibility. A bucket transformation distributes values among a fixed number of buckets using a defined hashing procedure. A truncate transformation groups values by a specified width or prefix behavior. These transformations are part of the table specification, not merely conventions inferred from directory names.\n\nFor example, consider a table partitioned by the day of event_time. A predicate selecting events between two timestamps can be projected onto the relevant day partitions. Because a day partition covers a broader range than an arbitrary timestamp interval, the partition filter may be conservative: it selects all partitions that might contain matching rows. A residual predicate then filters individual records or finer-grained file structures to preserve correctness.\n\nPartition evolution allows the partition specification to change without rewriting all earlier data. A table might begin with monthly partitions and later switch to daily partitions as ingestion volume grows. Older files retain the identifier of the specification under which they were written, while new files use the new specification. Query planning evaluates each group according to its own layout. The table can consequently contain several historical partition schemes at once.\n\nThis is different from physically reorganizing old data. Changing a specification affects subsequent writes; it does not automatically improve the pruning granularity of existing files. Rewriting older data under the new specification is a separate operation. The distinction permits gradual migration but also means that the performance effect of partition evolution depends on the relative volume of old and new data.\n\nSorting is another independent layout property. A table may define a sort order that writers can use to cluster records within or across files, improving the usefulness of column bounds. Partitioning determines coarse grouping, while sorting affects locality within that grouping. Neither a declared sort order nor a bucket partition creates a universally enforced row index, and actual layout depends on writer behavior and engine support."
    },
    {
      "heading": "Transactions, snapshots, and concurrent commits",
      "text": "Iceberg implements atomic table changes by preparing new files and metadata before publishing a new authoritative metadata reference. A writer first reads a table state, computes its changes, and writes the required data and metadata files. It then attempts to commit against the state it originally observed. Readers see either the previously committed state or the newly committed state, not an intermediate collection of partially published changes.\n\nThe core coordination model is optimistic concurrency control. Writers perform much of their work without holding an exclusive lock over the table, then validate assumptions at commit time. If another writer has committed first, the operation may refresh the table state and retry. Some operations can reuse already written files during a retry; others must fail or be recomputed because their assumptions are no longer valid.\n\nTwo independent appends often compose cleanly because neither removes data introduced by the other. A delete, overwrite, or merge can require stronger conflict checks. For instance, a writer deleting rows matching a predicate must consider whether a concurrent writer added files that could contain additional matching rows. Merely publishing both file sets without validation could produce a result inconsistent with the intended isolation level.\n\nIceberg supports operation-specific validation associated with snapshot and serializable isolation. The details depend on the write operation and engine integration; “ACID support” should not be interpreted as an unrestricted promise that every engine exposes identical behavior. The ACID properties of the table arise from several layers: atomic metadata publication, consistent file interpretation, isolation checks, and durable storage of committed objects.\n\nA query ordinarily selects a snapshot and reads that stable state even if another writer commits while execution is underway. This is closely related to multiversion concurrency control in databases. Historical snapshots can also be selected explicitly for time-travel queries. A timestamp-based lookup resolves to an appropriate recorded snapshot; it does not reconstruct arbitrary uncommitted or already expired states.\n\nThese guarantees are principally table-scoped. A Spark job updating two separate Iceberg tables does not automatically obtain a single atomic transaction covering both tables. Similarly, a successful commit to Iceberg does not inherently coordinate an external message acknowledgment or an update in another database. Broader transaction guarantees require additional protocols or services. Distinguishing table atomicity from application-wide atomicity is essential when interpreting the format’s transactional capabilities."
    },
    {
      "heading": "Query planning and selective reading",
      "text": "An Iceberg scan begins by resolving the table and selecting a snapshot. The planner then traverses the snapshot’s manifest list and relevant manifests to construct scan tasks. Because table membership is explicit, this process does not ordinarily require recursive enumeration of all storage directories. The engine obtains a defined set of candidate data files, together with the information needed to interpret them.\n\nFiltering proceeds at several levels. Partition summaries can eliminate entire manifests, partition values can eliminate file entries, and file-level column metrics can rule out additional files. Metrics can include lower and upper bounds, null counts, and value counts, depending on the column type and metrics configuration. For a predicate such as price > 100, a file whose recorded maximum is at most 100 can be excluded without reading its records.\n\nThese decisions must remain conservative. If metadata is missing, truncated, or insufficient to prove that a file cannot match, the planner retains the file. Different transformations and expression forms permit different degrees of pruning. The purpose is to avoid false exclusions: reading an unnecessary file costs time, whereas excluding a file containing matching records produces an incorrect answer.\n\nWithin retained files, the underlying format supplies further opportunities. Parquet row-group statistics and column projection can reduce the bytes read and decoded. A query optimizer may also push filters and projections through its Iceberg connector. These mechanisms are related but distinct: Iceberg prunes table files, while the file reader interprets file-internal metadata and encoding. The final record-level predicate remains necessary whenever earlier pruning is not exact.\n\nDistributed engines divide planned work into tasks that may process files or file ranges in parallel. Planning costs depend on manifest organization, metadata caching, file counts, and the amount of relevant delete information. A table containing billions of rows in a moderate number of well-sized files can be easier to plan than a much smaller table fragmented across millions of tiny files.\n\nIceberg exposes metadata tables in several engine integrations, allowing inspection of snapshots, history, manifests, files, and partitions through SQL or engine APIs. These views make it possible to examine file-size distributions, snapshot ancestry, and partition growth without treating storage listings as authoritative. Their exact names and available columns vary across integrations, and they describe table organization rather than supplying a complete record of every external access."
    },
    {
      "heading": "Row-level changes and streaming ingestion",
      "text": "Iceberg supports both append-oriented workloads and row-level modifications, but the physical cost of a modification depends on how it is represented. In a copy-on-write strategy, changing a subset of rows causes affected data files to be read and replaced with new files. This can make subsequent reads straightforward, but a small logical update may rewrite a much larger amount of data.\n\nA merge-on-read strategy records changes separately and applies them while reading. Iceberg format version 2 introduced standardized row-level delete files. Position deletes identify records by a data-file path and a zero-based row position. Equality deletes identify records through specified field values, allowing matching rows in applicable data files to be excluded without listing their individual positions.\n\nDelete application is governed by format rules, including sequence information and the relationships among data and delete files. A delete file is not simply a predicate that applies indiscriminately to every file ever added to a table. Ordering rules help distinguish data that existed when a delete was introduced from data added later. Implementations must preserve these semantics when rewriting files or combining changes.\n\nAn update can be implemented as removing the old row and inserting its replacement. SQL operations such as MERGE INTO are translated by the engine into supported Iceberg write operations; their availability and physical strategies depend on the integration. Equality deletes also do not establish uniqueness. If several applicable records share the specified values, the delete can affect all of them rather than an assumed single primary-key row.\n\nFor stream processing, engines can periodically commit batches of newly produced files as snapshots. Apache Flink and Spark Structured Streaming integrations support streaming use cases, but end-to-end delivery semantics depend on checkpointing, commit coordination, recovery, and source behavior. An Iceberg table’s atomic commit mechanism is one component of an exactly-once pipeline, not a substitute for those other mechanisms.\n\nIncremental reading requires similar care. Scanning files added between snapshots works naturally for append-oriented histories. It is not automatically equivalent to a complete change data capture stream containing every logical insert, update, and delete. Rewrites may replace files without changing the logical rows, while row-level deletes may require explicit interpretation. Consumers must distinguish physical file changes from business-level record changes and use the capabilities exposed by their engine accordingly."
    },
    {
      "heading": "Catalogs, interoperability, and access boundaries",
      "text": "A catalog maps table identifiers to the metadata needed to load and update tables. It commonly provides namespaces, table creation and deletion operations, and a mechanism for committing a new metadata location. This role overlaps with, but is narrower than, a general data catalog used for discovery, ownership, documentation, and governance. A deployment may integrate both functions, but they are not interchangeable concepts.\n\nIceberg supports multiple catalog implementations. Hive-backed catalogs use a Hive Metastore; JDBC-backed catalogs use a relational database; other implementations integrate with cloud services or specialized metadata systems. The REST catalog specification defines an HTTP-based interface intended to decouple clients from catalog internals. An engine can communicate through a standardized protocol rather than embedding every backend’s native catalog logic.\n\nAtomic commit behavior depends on the catalog and storage combination. Some implementations use a conditional metadata update or compare-and-swap operation; others rely on locking or storage primitives appropriate to their environment. File-system-oriented mechanisms that assume atomic rename cannot simply be transferred to every object store. Correct deployment therefore requires more than placing metadata files in a shared bucket.\n\nInteroperability has several layers. Two engines may both recognize Iceberg yet differ in supported format versions, data types, delete encodings, catalog authentication, SQL syntax, or write operations. Reading ordinary append-only Parquet-backed tables is a less demanding compatibility test than concurrently writing tables with schema evolution and row-level deletes. Format compatibility and feature compatibility must be evaluated separately.\n\nSecurity is also layered. Catalog services can enforce authentication and authorize metadata operations, while storage systems enforce access to data and metadata objects. A REST catalog may participate in distributing scoped storage credentials where the relevant implementation supports that mechanism. However, users holding broad direct storage permissions may be able to bypass restrictions enforced only at the query or catalog layer.\n\nThe public format does not itself mandate a universal row-level security or column-masking policy. Those controls are generally implemented by query engines and governance services, and consistent enforcement across multiple engines requires coordinated integration. Table portability consequently does not imply automatic portability of all security policies, audit records, or business semantics. Shared files make cross-engine access possible; shared operational rules make that access coherent."
    },
    {
      "heading": "Maintenance, retention, and physical storage costs",
      "text": "Iceberg’s immutable-file model accumulates historical metadata and, potentially, old data files. Snapshot expiration removes selected historical snapshots from the table’s retained history and can make files eligible for deletion when no remaining reference needs them. Expiration differs from deleting current rows: it changes how much table history remains available rather than directly defining the current logical contents.\n\nBranches and tags extend retention beyond a single current snapshot. A branch is a named reference that can advance through commits, while a tag identifies a particular snapshot. These mechanisms support workflows such as retaining a known historical state or isolating a sequence of writes. They are snapshot references, not automatically equivalent to the conflict resolution and arbitrary content merging provided by a source-code version control system.\n\nRetention must account for active readers and external consumers. A long-running query may still require files from the snapshot it selected at startup. If those files are physically removed before the query finishes, snapshot consistency alone cannot keep the read working. Retention windows, reference policies, and maintenance scheduling therefore interact with expected query durations and recovery requirements.\n\nOrphan-file removal addresses a different problem. Writers can create files and then fail before committing them, leaving objects that are not referenced by table metadata. Identifying such objects generally requires comparing storage contents with the files reachable through metadata. Files that appear unreferenced may still belong to in-progress writes, so age thresholds and correct path handling are important to avoid deleting legitimate work.\n\nCompaction rewrites many small files into fewer larger files, potentially improving scan efficiency and reducing scheduling overhead. It can also incorporate applicable deletes into replacement data files. Manifest rewriting reorganizes metadata rather than table rows, improving planning when manifests are numerous or poorly aligned with query patterns. Neither operation is free: both consume compute and storage I/O and may temporarily require additional capacity.\n\nThe economics of maintenance follow directly from workload shape. Frequent small streaming commits create more snapshots and often more small files; update-heavy workloads can accumulate delete information or repeated rewrites. Retaining extensive history increases storage requirements, especially when old files remain reachable after substantial changes. Time travel is therefore a retention feature with measurable costs, not an unlimited historical archive, and it does not replace independent backup or disaster-recovery arrangements."
    },
    {
      "heading": "Format versions, ecosystem, and practical limits",
      "text": "Iceberg separates the evolution of its on-disk specification from software release numbering. Format version 1 established the foundational snapshot, manifest, and schema model. Version 2 added facilities needed for standardized row-level deletes and related sequencing. Version 3 extends the specification further, including deletion vectors and row-lineage facilities. Support for a particular format version or feature must be distinguished from the version number of an engine connector or library.\n\nA format upgrade can affect the set of compatible readers and writers. An engine that understands only an older format may be unable to interpret newly introduced metadata or delete representations correctly. This makes upgrades a coordination issue in environments with multiple independent clients. Public specifications enable interoperability, but they do not make unsupported features intelligible to older software.\n\nIceberg is often compared with Delta Lake and Apache Hudi, which also organize analytical tables over file or object storage. Their metadata structures, change representations, indexing facilities, catalog relationships, and engine integrations differ. All three address problems beyond what a standalone columnar file format solves, but they are not interchangeable encodings of the same table. Moving between them can require rebuilding metadata, changing operational workflows, or rewriting data.\n\nThe distinction from a data warehouse is equally important. Iceberg specifies a storage-facing table layer; it does not by itself supply a cost-based optimizer, workload manager, SQL server, user interface, or complete governance system. Commercial and open-source systems can combine it with these components. Conversely, adopting Iceberg files does not guarantee warehouse-like response times for every workload.\n\nIts strengths are particularly relevant to large scans, evolving analytical datasets, and cross-engine access. Highly selective transactional workloads have different requirements: frequent single-row lookups, enforced uniqueness, low-latency commits, and large numbers of concurrent tiny transactions may favor a purpose-built relational database or another serving system. Iceberg can participate in pipelines feeding those systems without replacing their operational role.\n\nPerformance and correctness remain properties of the full deployment. Poor file sizing, excessive metadata growth, incompatible writers, weak commit coordination, or inconsistent access policies can undermine the advantages of the format. Evaluating an Iceberg system therefore requires examining actual query patterns, write conflicts, supported features, storage behavior, and maintenance operations rather than treating the presence of an open table format as a complete architectural guarantee."
    }
  ],
  "links": [
    "open-source-software",
    "apache-parquet",
    "database-management-system",
    "data-lake",
    "data-lakehouse",
    "metadata",
    "apache-license",
    "file-system",
    "apache-spark",
    "cloud-computing",
    "data-compression",
    "b-tree",
    "database-schema",
    "primary-key",
    "acid-properties",
    "database-transaction",
    "query-optimizer",
    "sql",
    "stream-processing",
    "change-data-capture",
    "data-catalog",
    "authentication",
    "version-control",
    "data-backup",
    "data-warehouse",
    "relational-database"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}