{
  "id": "data-lakehouse",
  "title": "Data lakehouse",
  "lang": "en",
  "source_lang": "en",
  "summary": "A data architecture that combines data-lake storage with transactional reliability, governance, and analytical capabilities associated with data warehouses.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "Lakehouse",
    "data lakehouse architecture",
    "lakehouse architecture"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/data-lakehouse/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/data-lakehouse/",
      "json": "https://aiwiki.page/zh/data-lakehouse.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A data lakehouse is a software architecture for analytical data management that combines the flexible, relatively inexpensive storage of a data lake with capabilities traditionally associated with a data warehouse. These capabilities include reliable table updates, controlled schemas, catalog-based discovery, access policies, and efficient analytical queries. A lakehouse typically stores data in files on shared storage and uses a transactional table layer to present those files as managed tables.\n\nThe architecture seeks to let business reporting, exploratory analysis, and machine learning operate on a common foundation rather than on separately maintained lake and warehouse copies. Its defining idea is not simply that both kinds of system exist together, but that warehouse-like management and performance can be provided directly over lake storage. The term describes an architectural approach rather than a single product, file format, or formal standard; implementations differ substantially in their transaction guarantees, supported workloads, and degree of openness."
    },
    {
      "heading": "Origins and architectural boundaries",
      "text": "Data lakehouses emerged from the convergence of several earlier developments. Data warehouses had established integrated, curated repositories for reporting and decision support, commonly using SQL, dimensional models, and tightly managed loading processes. During the 2000s and 2010s, Apache Hadoop and later cloud storage made it practical to retain large volumes of logs, event records, and less structured data outside a conventional warehouse. These data lakes offered flexibility but often lacked consistent mechanisms for managing concurrent changes, discovering trustworthy datasets, or enforcing a shared interpretation of their contents.\n\nA widely used response was a two-tier arrangement: retain raw data in a lake, then transform and copy selected datasets into a warehouse. This separated experimental processing from curated reporting, but it also created synchronization work. A correction might have to pass through several pipelines before appearing in reports, while the lake and warehouse could temporarily contain different versions of the same business facts. Multiple copies also required separate security configuration, reconciliation, and storage management. These problems were not inevitable in every deployment, but they supplied much of the motivation for a unified architecture.\n\nThe label became prominent around 2020 and was articulated in the 2021 CIDR paper Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics, by Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia. The paper argued for combining warehouse-style management and performance with direct access to data in open formats. The term was strongly associated with Databricks, but the architectural pattern extends beyond that vendor. Its technical foundations include independently developed table formats and query engines, rather than a single coordinated invention.\n\nThe boundary between a lakehouse and neighboring architectures is consequently imprecise. Adding an SQL engine to a collection of files does not by itself supply transactional reliability or coherent table management. Conversely, a modern cloud warehouse may separate storage from computation, support external tables, and process semi-structured data, giving it features commonly associated with lakehouses. “Lakehouse” is therefore more informative when accompanied by concrete statements about where authoritative data resides, how tables are committed, which engines can read and write them, and where policies are enforced.\n\nThe concept also concerns analytical rather than transactional application workloads. A lakehouse can support updates and deletions without becoming a substitute for every operational database. Processing millions of application requests through low-latency key lookups and small transactions imposes different requirements from scanning historical records for an analytical query. Operational databases frequently remain upstream systems of record, while the lakehouse holds replicated or transformed representations for analysis."
    },
    {
      "heading": "Storage, tables, catalogs, and compute",
      "text": "A typical lakehouse separates durable storage, table management, and computation. The storage layer commonly uses object storage, such as Amazon S3, Azure Blob Storage, or Google Cloud Storage, although distributed file systems can serve the same architectural role. Object stores expose named objects rather than the full behavior of a local disk. They provide scalable persistence, but they do not automatically turn a directory of related files into a transactional table. That additional abstraction belongs to the table-management layer.\n\nAnalytical records are frequently stored in a column-oriented file format such as Apache Parquet. A columnar layout groups values from the same column, enabling queries to avoid reading unrelated fields and allowing effective data compression. A query that calculates revenue by region need not read large text descriptions stored in other columns. Files also contain structural information and statistics that readers can use to locate relevant data. Different file formats can coexist within the wider platform, but an individual table format generally specifies which arrangements it supports.\n\nThe table layer defines which files constitute a particular table version and how changes become visible. Its metadata can describe schemas, partition organization, file-level statistics, delete information, and references to historical states. This is a crucial distinction: a table is not necessarily the set of every file currently present under a storage prefix. Some objects may belong to older versions, represent incomplete writes, or await cleanup. Readers following the table protocol can identify the valid state without interpreting every storage object independently.\n\nA data catalog supplies a naming and discovery layer. Users refer to tables through names organized into namespaces rather than manually locating file paths. Depending on the implementation, a catalog may also participate in atomic commits, resolve a table’s current metadata location, manage permissions, or record ownership. The catalog is logically distinct from the table format: a format specifies how a table is represented and changed, while a catalog specifies how named tables are located and, in some designs, coordinated.\n\nCompute engines read the table metadata and schedule work across processors or machines. Examples include Apache Spark, Trino, and warehouse-oriented engines that support external table formats. Because storage persists independently of a particular compute cluster, separate engines can be provisioned for transformation jobs, interactive analysis, or model preparation. This separation makes resource allocation more flexible, but shared storage alone does not guarantee safe interoperability. Engines must agree on table semantics, supported features, and commit behavior; otherwise, two programs accessing the same files can still obtain incompatible results."
    },
    {
      "heading": "Transactional consistency and table evolution",
      "text": "Reliable modification of shared files is a central lakehouse problem. Consider an update that replaces ten data files with twelve new ones. If readers simply list a directory while the update is underway, they could encounter a mixture of old and new files. A transactional table layer instead publishes a new logical state only after its necessary components have been prepared. Readers continue using the preceding state until the commit becomes visible, preventing a partially completed rewrite from appearing as a valid table.\n\nThese mechanisms implement database transactions and, to varying extents, the ACID properties of atomicity, consistency, isolation, and durability. Atomicity concerns whether a change becomes visible as a whole; durability concerns whether a committed state survives failures. Consistency includes adherence to enforced table rules, not a guarantee that every business fact is correct. Isolation determines how concurrent operations interact. A product’s claim to support ACID transactions therefore requires qualification: guarantees can differ by operation, isolation level, storage configuration, and whether the transaction covers one table or several.\n\nMany implementations use optimistic concurrency control. Writers perform work without holding a long-lived exclusive lock, then validate whether another commit has invalidated their assumptions. If two independent appends are compatible, both may succeed. If two writers rewrite overlapping data, one may need to retry. Snapshot-based reading resembles multiversion concurrency control: a reader uses a stable version while writers prepare newer states. The precise conflict rules matter, particularly when concurrent jobs combine inserts, updates, deletes, and schema changes.\n\nUpdates and deletions can be implemented in different ways. Copy-on-write rewrites affected data files so that subsequent readers see already-consolidated records. Merge-on-read retains base data together with additional change or deletion information that readers reconcile. These approaches trade write amplification against read complexity. An implementation may use position-based deletion records, key-based deletion records, or deletion vectors, depending on the table format and enabled features. None eliminates physical maintenance: rewritten files, accumulated changes, and obsolete metadata eventually require consolidation or cleanup.\n\nTable evolution adds another dimension. A schema can change as columns are introduced, renamed, or assigned new types. Stable internal field identifiers can distinguish a genuine rename from dropping one column and creating another, but allowed changes remain format- and engine-dependent. Historical snapshots can support time-travel queries and reproducible analysis, provided the referenced files are retained. Expiring snapshots or deleting obsolete files removes some of that history. Likewise, transactional support within individual tables does not automatically provide atomic multi-table business operations, enforced foreign keys, or global uniqueness across concurrent ingestion jobs."
    },
    {
      "heading": "Open table formats and interoperability",
      "text": "Several open table formats provide important foundations for lakehouse implementations. Delta Lake, Apache Iceberg, and Apache Hudi all manage analytical tables over file-based storage, but they have different metadata structures, feature histories, and operational emphases. They should not be confused with file formats. Parquet describes the organization of records within a file; a table format describes the relationship among files, schemas, versions, and changes. A Parquet reader alone cannot necessarily interpret a complete table containing deleted rows, historical files, or concurrent commits.\n\nDelta Lake was released as open-source software in 2019. Its table state is represented through a transaction log, conventionally stored in a _delta_log directory alongside the data. Log actions describe changes such as adding and removing files or updating metadata, while checkpoints accelerate reconstruction of table state. Delta’s protocol includes versioning and feature negotiation, allowing the format to evolve while specifying what a reader or writer must understand. Actual compatibility depends on the enabled table features, not merely on whether an engine advertises basic Delta support.\n\nApache Iceberg originated at Netflix and was open-sourced in 2018. Its metadata hierarchy uses table metadata, snapshots, manifest lists, and manifests to identify the files belonging to a table state. Iceberg emphasizes features including field identifiers and partition evolution. In its hidden-partitioning model, users can filter on logical columns while the engine uses declared partition transforms to eliminate irrelevant data. A catalog commonly helps coordinate the update that makes a new metadata state current, although the details depend on the catalog implementation.\n\nApache Hudi originated at Uber in 2016 and became an Apache top-level project in 2020. It places particular emphasis on incremental ingestion, updates, and efficient record-level change handling. Its timeline records table actions, and its copy-on-write and merge-on-read table types support different balances between ingestion cost and query work. Indexing mechanisms help locate records affected by updates. Hudi also supports incremental consumption, allowing downstream processes to identify changes rather than repeatedly scan an entire dataset.\n\nAll three are open-source projects, but openness is not a binary guarantee of interchangeability. One engine may read a table correctly yet lack support for writing its latest deletion representation, evolving its partition layout, or consuming its change feed. Catalog interfaces, authorization systems, and maintenance procedures may also differ. A useful interoperability assessment therefore distinguishes file readability, table-protocol compatibility, read-write support, and operational compatibility. Conversion or translation between formats can broaden access, but the treatment of snapshots, deletions, and simultaneous writers must remain explicit."
    },
    {
      "heading": "Query processing and performance",
      "text": "A lakehouse must do more than preserve correct table states: it must also make large-scale analysis efficient. Analytical performance depends on reducing the data read, choosing suitable execution strategies, and moving intermediate results economically. A query optimizer translates SQL into an execution plan using information such as row counts, value distributions, partition layout, and available processing resources. Table metadata and file statistics extend this optimization beyond the contents of any single file.\n\nPartition pruning eliminates groups of files whose partition values cannot satisfy a query. Data skipping can use minimum and maximum values or other statistics to exclude additional files. Predicate pushdown applies supported filters close to the storage reader, while column pruning avoids reading unused fields. For example, a query restricted to one month of transactions and three columns should ideally avoid both unrelated months and unrelated columns. These mechanisms are complementary, and their effectiveness depends on how the physical layout corresponds to common filters.\n\nLayout design involves competing demands. Partitioning by a high-cardinality field can produce huge numbers of small directories or files; partitioning too coarsely leaves queries scanning excessive data. Sorting or clustering records within files can improve skipping for frequently filtered columns without requiring a partition for each value. Unlike a conventional database index, such organization may not provide direct record lookup, but it can sharply reduce the fraction of files examined. Format-specific partition evolution can allow new layout choices without rewriting all historical data immediately.\n\nExecution engines use parallel computing and distributed computing to divide scans, joins, and aggregations among workers. Vectorized operators process batches of values rather than interpreting one row at a time. Local caches can reduce repeated remote reads, while broadcast joins avoid redistributing a large fact table when the other input is small enough to replicate. Conversely, large shuffles, skewed join keys, insufficient memory, and inaccurate statistics can dominate runtime regardless of the openness or reliability of the storage layer.\n\nSmall files are a recurring operational problem because each file adds metadata processing, request overhead, and task scheduling work. Compaction combines files into more efficient units, but excessively large files can reduce parallelism or increase the amount rewritten by updates. Performance comparisons must therefore control for dataset layout, caching, concurrency, refresh rate, and maintenance work, not just query text. A fast isolated benchmark does not establish equally low latency under mixed ingestion and reporting loads. Nor does separation of storage and compute remove storage bandwidth limits or coordination overhead."
    },
    {
      "heading": "Ingestion, transformation, and analytical workloads",
      "text": "Lakehouse ingestion ranges from periodic bulk loading to continuous event processing. Bulk jobs can copy source extracts into staging tables and then publish validated results. Streaming pipelines process records as they arrive, while change data capture transfers inserts, updates, and deletes from operational databases. The source’s ordering rules and transaction boundaries do not automatically survive transport. Connectors and transformation jobs must interpret identifiers, timestamps, delete markers, and retries consistently to construct the intended analytical state.\n\nA common organizational pattern divides data into bronze, silver, and gold layers, often called a medallion architecture. Bronze retains source-oriented records, silver applies cleaning and integration, and gold provides business-focused tables or aggregates. These labels are conventions rather than required lakehouse components. Their value lies in making responsibilities explicit: preserving enough source information for replay, producing reusable cleaned data, and maintaining clearly defined analytical products. Several physical copies can still exist within such a design, so a lakehouse does not literally imply storing each fact only once.\n\nStream processing introduces distinctions between event time, processing time, and commit time. An event may describe a purchase made at noon but arrive much later because of a disconnected device or delayed upstream service. Pipelines may therefore revise previously published totals. Checkpoints and transactional sink commits can prevent some duplicate effects after failures, but an “exactly once” claim depends on the complete path, including the source, state management, and sink. An external side effect, such as sending a message, requires its own coordination if it must match a table commit.\n\nBusiness intelligence typically consumes curated relational tables through SQL, whereas data scientists may use data frames, Python libraries, and distributed computation. A shared storage foundation can reduce the need to export warehouse data before preparing training data. However, a reliable machine-learning dataset also requires point-in-time correctness. Joining historical examples to information that became available only later introduces data leakage, even if every table is transactionally consistent. Snapshot versions help identify the inputs used, but they do not by themselves establish correct temporal joins.\n\nLakehouses can also organize image references, documents, audio metadata, and other less structured assets. In many deployments, the binary objects remain in storage while tables hold locations, labels, extracted attributes, or processing status. This is different from storing every asset directly as a conventional table cell. Search indexes, vector indexes, feature-serving systems, and operational databases may still be built downstream for specialized access patterns. Their existence is compatible with the architecture, provided the relationship between authoritative tables and derived serving copies is maintained."
    },
    {
      "heading": "Governance, security, and data reliability",
      "text": "Centralized storage creates opportunities for unified governance, but it also increases the consequences of poorly controlled access. Access control may apply at several levels: cloud accounts and storage containers, catalogs and schemas, tables, columns, or individual rows. A query engine can enforce a masking rule while returning results, yet a user with direct permission to read the underlying objects may bypass that rule. Effective enforcement therefore depends on the complete access path, not solely on the permissions visible in an SQL interface.\n\nAuthentication establishes the identity of a person or service, while authorization determines which operations that identity may perform. Applying least privilege involves separating routine readers, ingestion services, table maintainers, and administrators where appropriate. Encryption protects data in transit and at rest, but does not substitute for these distinctions. A service with permission to decrypt and read every file can still expose data through an overly broad query endpoint or an improperly configured export process.\n\nCatalogs and data lineage help explain what datasets mean and how they were produced. Lineage can connect a report column to a curated table, transformation job, and source field. Ownership records, descriptions, sensitivity labels, and versioned pipeline definitions make this information operationally useful. Automatic lineage remains incomplete when code performs opaque operations, reads files outside managed interfaces, or transfers data into external tools. A lakehouse can supply collection points for governance metadata without ensuring that every dependency is captured.\n\nData quality is distinct from transactional consistency. A committed table can contain duplicate customers, invalid units, missing records, or contradictory business definitions. Schema checks catch some structural problems, while additional rules can test ranges, uniqueness, referential relationships, and reconciliation totals. A rejected batch may be quarantined for investigation rather than silently discarded. Such mechanisms belong partly to ingestion and orchestration layers; they are not uniformly guaranteed by the underlying table format.\n\nRetention and deletion present a particularly important interaction between governance and versioned storage. Removing a row from the latest snapshot may leave its values in older files, backups, replicas, or downstream exports. Conversely, deleting historical files too soon can break long-running queries or eliminate evidence needed for reproducibility. Policies must distinguish logical deletion, snapshot expiration, physical object removal, and backup retention. Data privacy obligations and audit requirements therefore interact with technical maintenance settings, rather than being satisfied merely by issuing a SQL delete statement."
    },
    {
      "heading": "Economics, operations, and architectural trade-offs",
      "text": "The economic argument for a lakehouse usually begins with shared, scalable storage and independently provisioned compute. A large historical dataset can remain available while different workloads use different engines or resource sizes. Idle compute may be stopped without deleting the data, and expensive processing capacity can be reserved for periods when it is needed. These properties are attractive, but they are not unique to lakehouses: many cloud warehouses also separate storage and compute. The relevant comparison is the complete deployment rather than the architectural label.\n\nTotal cost includes storage, compute, object requests, network transfer, catalogs, orchestration, monitoring, and engineering effort. Retained snapshots and intermediate datasets can increase storage consumption substantially above the size of the latest logical tables. Compaction and clustering consume resources even when they do not produce new business records. Frequent metadata operations can matter for tables with many files, while movement across regions or cloud providers may incur network charges. Managed services shift some operational work to a vendor, but their pricing and workload controls become part of the same calculation.\n\nMaintenance is an ongoing part of the system. Operators must manage file sizes, obsolete snapshots, orphaned objects, statistics, ingestion backlogs, and failed commits. Cleanup procedures need safeguards because an object that appears unused may still be referenced by a retained snapshot or an active writer. Recovery also requires more than durable data files: catalogs, credentials, policies, pipeline definitions, and metadata histories can be essential. Backups and disaster recovery plans must preserve a consistent relationship among these components.\n\nWorkload isolation is another practical constraint. Separate compute clusters can reduce direct competition for processors and memory, but they may still share storage bandwidth, catalog capacity, commit coordination, and table-maintenance work. A continuous writer and a large merge job can conflict even when they run on different machines. Service objectives therefore need to distinguish query latency, ingestion freshness, availability, and recovery time. A configuration optimized for hourly reporting may be unsuitable for a dashboard requiring continuously updated results.\n\nOpen formats can reduce dependence on a particular compute engine, but they do not automatically eliminate vendor dependence. Proprietary catalog functions, security policies, optimizer behavior, orchestration services, or unsupported table features can make migration difficult even when the underlying data files remain readable. Equally, adopting many interchangeable components can increase integration and operational complexity. Architectural evaluation consequently turns on concrete requirements: supported transaction scope, acceptable freshness, compatibility across actual readers and writers, maintenance burden, and performance under representative concurrent workloads."
    }
  ],
  "links": [
    "software-architecture",
    "data-management",
    "data-lake",
    "data-warehouse",
    "machine-learning",
    "sql",
    "apache-hadoop",
    "database",
    "file-system",
    "apache-parquet",
    "data-compression",
    "metadata",
    "data-catalog",
    "apache-spark",
    "database-transaction",
    "acid-properties",
    "database-schema",
    "apache-iceberg",
    "open-source-software",
    "query-optimizer",
    "database-index",
    "parallel-computing",
    "distributed-computing",
    "change-data-capture",
    "stream-processing",
    "training-data",
    "data-leakage",
    "access-control",
    "authentication",
    "least-privilege",
    "encryption",
    "data-lineage",
    "data-quality",
    "data-privacy",
    "data-backup",
    "disaster-recovery"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}