{
  "id": "apache-parquet",
  "title": "Apache Parquet",
  "lang": "en",
  "source_lang": "en",
  "summary": "Apache Parquet is an open column-oriented file format for efficient storage and analytical processing of structured and nested data.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "Parquet",
    "Apache Parquet file format"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/apache-parquet/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/apache-parquet/",
      "json": "https://aiwiki.page/zh/apache-parquet.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Apache Parquet is an open, column-oriented binary file format designed for storing and retrieving large analytical datasets. It organizes values by column within groups of rows, allowing software to read selected fields without reading every field in each record. Its combination of column-specific encodings, compression, and descriptive metadata makes it widely used in data warehouses, data lakes, and distributed query systems.\n\nParquet is a storage format rather than a database management system: it defines how data and associated structural information are represented in files, but does not itself execute queries, coordinate concurrent writers, or maintain transactions across a collection of files. Its importance lies in providing a shared representation that many independent systems can produce and consume. The format supports primitive values, nested records, optional fields, and repeated structures, while allowing readers to skip portions of data that cannot contribute to a query.\n\nOriginally developed through collaboration between Twitter and Cloudera and publicly released in 2013, Parquet became an Apache Software Foundation top-level project in 2015. Its design combines ideas from column-oriented analytical databases with techniques for representing nested data introduced by Google’s Dremel system."
    },
    {
      "heading": "Origins and design objectives",
      "text": "Parquet emerged as organizations increasingly used Apache Hadoop to retain large collections of event logs, business records, and other analytical data. Early Hadoop pipelines often stored records as text or used formats oriented toward reading entire records. These approaches were convenient for ingestion and record-by-record processing, but inefficient when an analytical query needed only a few fields from a wide dataset. Reading a user identifier and a purchase amount should not require decoding dozens of unrelated attributes.\n\nThe initial collaboration between Twitter and Cloudera addressed this mismatch by combining a column-oriented organization with support for complex, nested records. Julien Le Dem and other contributors helped develop the project, which was released in 2013. Parquet entered the Apache Incubator in 2014 and became a top-level Apache project in April 2015. Its development under the Apache License 2.0 encouraged adoption across commercial products and open-source software without tying the format to a single execution engine.\n\nAn important intellectual predecessor was Dremel, described by Google researchers in 2010. Dremel demonstrated how nested records could be decomposed into columns while preserving the information needed to reconstruct their structure. Parquet adopted the central technique of definition and repetition levels, making it possible to represent optional and repeated values without storing every nested object as an opaque serialized object. This was particularly relevant to event data, where a record might contain an optional user profile and a variable-length list of interactions.\n\nParquet’s design separates the storage representation from the data-processing framework. Although it developed within the Hadoop ecosystem, a Parquet file does not require Hadoop to be read. The same file can be accessed from a local file system, a distributed file system, or cloud object storage, provided the reader can obtain the required byte ranges. This separation helped Parquet spread into desktop analytical tools, cloud warehouses, and services built around Apache Spark.\n\nThe format also separates encoding from general-purpose compression. A column of repeated country codes can first be represented as dictionary identifiers, then compressed with a codec. A column of ordered integers may instead benefit from delta encoding. This modularity allows writers to adapt to data characteristics without changing the application’s logical schema. The guiding objective is not universally minimal file size, but a useful balance among storage space, selective reading, decoding cost, and interoperability."
    },
    {
      "heading": "File organization and metadata",
      "text": "An ordinary, unencrypted Parquet file starts with the four-byte marker PAR1, followed by encoded data and supporting structures. Near the end is serialized file metadata, followed by a four-byte little-endian metadata length and another PAR1 marker. This arrangement lets a reader inspect the file’s tail, locate the footer, and discover the schema and data locations before requesting the relevant portions of the file. Parquet’s metadata structures are defined using Apache Thrift and serialized with its compact protocol.\n\nThe principal organizational hierarchy is file, row group, column chunk, and page. A row group contains a horizontal subset of the dataset: for example, a consecutive batch of several hundred thousand records. Within that row group, each leaf column has a column chunk containing the corresponding values and structural information. A column chunk consists of pages, which are the basic units used for encoding and compression. For nested data, a leaf column is a primitive field reached through the nested schema, rather than necessarily a top-level application field.\n\nThe file footer describes the schema and row groups, including column metadata, locations, encodings, compression codecs, and statistics where available. These descriptions allow a reader to identify which byte ranges correspond to the fields it needs. The term “columnar” therefore does not mean that every value of one column occupies a single uninterrupted region across the entire file. A column normally has a separate chunk in each row group, preserving a useful compromise between column-wise access and manageable processing units.\n\nPages carry their own headers and have several roles. Data pages contain encoded values and, where required, definition and repetition levels. A dictionary page supplies the values referenced by dictionary-encoded data pages in its column chunk. Optional page indexes provide statistics and offsets that support finer-grained skipping. Page boundaries are generally independent across columns, so page number three in one column does not necessarily cover the same records as page number three in another.\n\nFooter-oriented discovery has operational consequences. Readers generally need seek-like or range-read access, and a file is not a simple stream of independently complete records. A writer normally finalizes the footer when it closes the file; an interrupted write that lacks a usable footer may leave an unreadable result. Concatenating two complete Parquet files byte for byte does not create one valid combined dataset file. Dataset-level aggregation is ordinarily achieved by managing multiple files, or by reading and rewriting their contents into a new file."
    },
    {
      "heading": "Physical types, logical types, and schemas",
      "text": "Parquet distinguishes the physical representation of values from their logical meaning. Its basic physical types are BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BYTE_ARRAY, and FIXED_LEN_BYTE_ARRAY. These types describe the storage-level form. Logical type annotations specify how a value should be interpreted, allowing the same physical representation to express several application-level concepts. This distinction is central to interoperability because programming environments do not all use the same native types.\n\nFor example, a date can be represented as an INT32 containing the number of days relative to the Unix epoch, with a date annotation identifying its meaning. A timestamp can use an INT64 with a specified time unit and an indication of whether it is adjusted to UTC. A string is commonly stored as a BYTE_ARRAY annotated as a string and encoded using UTF-8. Without the appropriate annotation, a reader may correctly recover the bytes while still being unable to determine whether they represent text, an identifier, or arbitrary binary content.\n\nDecimal values illustrate how logical precision differs from physical width. Parquet’s decimal annotation specifies precision and scale, with the represented number interpreted as\n\n\\[x = u \\times 10^{-s},\\]\n\nwhere \\(u\\) is the stored unscaled integer and \\(s\\) is the scale. Depending on precision and implementation, decimals may use integer or byte-array physical types. Their representation is distinct from binary floating-point, whose rounding properties follow different rules. The FLOAT and DOUBLE types correspond to binary floating-point formats associated with IEEE 754.\n\nA Parquet schema also describes groups and field repetition: fields may be required, optional, or repeated. Logical annotations define conventions for structures such as lists and maps. Standard list representation uses nested groups to distinguish an absent list, an empty list, and a list containing null elements. Historical files sometimes use older list encodings, requiring readers to apply compatibility rules. Schemas can also carry field identifiers, which some table systems use to preserve field identity independently of a field’s current name or position.\n\nThe schema embedded in one file does not automatically establish a dataset-wide schema. If different files contain different fields or types, the reading system must decide how to reconcile them. Adding a nullable column is often straightforward because older files can supply null for that field, but renaming a field or changing timestamp semantics requires additional conventions. The legacy INT96 type is a prominent compatibility case: it has historically been used for timestamps, but lacks the clear, standardized logical interpretation of modern timestamp annotations. Accordingly, physical readability alone does not guarantee semantic agreement between systems."
    },
    {
      "heading": "Representing nested and missing values",
      "text": "Parquet’s handling of nested data is one of its defining features. Consider records containing an optional address group with an optional city string. A missing city may mean that the entire address is absent, or that the address exists but its city field is absent. Simply storing a sequence of strings and null markers would lose this distinction. Parquet attaches definition levels to leaf-column entries to record how much of the path through the schema is present.\n\nThe maximum definition level for a leaf is determined by the optional and repeated nodes along its schema path. Each such node contributes a level at which data can be undefined. A level below the maximum indicates where the path ceased to be defined, while a maximum level indicates that an actual primitive value is present. In the optional-address, optional-city example, a definition level of zero can indicate no address, one an address without a city, and two a present city value. Only entries at the maximum definition level consume a value from the encoded primitive-value stream.\n\nRepeated structures require another signal because a leaf column may contain multiple entries for one logical record. Repetition levels identify the repeated depth at which a new entry continues an existing structure. A repetition level of zero marks the beginning of a new top-level record. Higher levels indicate continuation within repeated ancestors. Together, definition and repetition levels allow a reader to reconstruct nested records while reading their leaf columns separately.\n\nA simple example uses a required group tags containing a repeated string field tag. Suppose three records have tag lists [\"red\", \"blue\"], [], and [\"green\"]. The leaf-column entry sequence can be described by repetition levels \\(0,1,0,0\\), definition levels \\(1,1,0,1\\), and the physical values \"red\", \"blue\", and \"green\". The third entry records the empty list without contributing a string value. A nullable list whose elements may also be null needs a richer schema and additional definition levels to distinguish those cases.\n\nThese level streams are themselves encoded compactly, commonly using run-length and bit-packed techniques. Flat required fields need no definition or repetition information, so the mechanism does not impose identical overhead on every schema. Deeply nested structures, by contrast, can require more structural decoding and more complex reconstruction logic.\n\nThe practical benefit is selective access to nested leaves. A query computing the total price of items in an order need not necessarily load unrelated product descriptions or shipping-address fields. Nevertheless, reconstructing relationships among several repeated columns requires respecting their structural levels; treating them as unrelated arrays can produce incorrect pairings. Nested columnar storage therefore saves input and output while retaining enough structure for correct record reconstruction."
    },
    {
      "heading": "Encodings and compression",
      "text": "Parquet reduces storage and reading costs through two related but distinct mechanisms: value encoding and lossless compression. Encoding exploits the structure of values within a column, whereas a compression codec operates on the resulting byte representation. The best combination depends on cardinality, ordering, value distribution, and the balance between processing cost and transferred bytes. A file may use different encodings and codecs for different columns, subject to the format’s rules and the writer’s capabilities.\n\nPlain encoding provides a direct baseline representation for each physical type. Dictionary encoding instead builds a set of distinct values and stores integer identifiers referring to that set. A column containing millions of entries drawn from a small collection of country codes can therefore store the country strings once in a dictionary and represent the data pages with compact identifiers. The dictionary belongs to a column chunk, not to the entire dataset. If a dictionary becomes too large or ceases to be beneficial, a writer can use another supported encoding for subsequent data pages.\n\nParquet’s run-length/bit-packing hybrid represents repeated integers as runs and packs small integers using only the required number of bits. It is used for dictionary identifiers and structural levels. Delta binary packing stores differences between integer values and packs them in blocks, making it useful when adjacent values vary predictably. Delta length byte-array encoding compresses sequences of lengths, while delta byte-array encoding also exploits shared prefixes between successive byte-array values. Their effectiveness therefore depends partly on the order in which records were written.\n\nByte-stream split is another encoding, particularly useful for certain fixed-width numerical data. It rearranges bytes so that bytes at the same position in successive values are grouped into separate streams. The transformation itself does not reduce the number of bytes, but can expose patterns that a downstream compressor handles more effectively. This illustrates why encoding and compression should not be treated as interchangeable terms: a representation change can improve compression without independently shrinking the data.\n\nParquet supports codecs including Snappy, Gzip, Zstandard, and Brotli, as well as uncompressed storage. Codec support varies among readers and library versions, and the distinction between older LZ4 handling and LZ4_RAW is relevant to compatibility. Compression is generally applied to page data rather than to the entire file as one stream, preserving the ability to access selected regions. Data Page V2 also treats level information differently from the original page layout: its level sections remain separate from the optionally compressed value section. Consequently, codec choice alone does not fully describe the on-disk representation or the cost of decoding it."
    },
    {
      "heading": "Selective reading and query execution",
      "text": "Parquet’s principal analytical advantage is avoiding unnecessary work. Column projection lets an engine read only the columns referenced by a query. For a table with hundreds of attributes, an aggregation over two numerical fields may require only a small fraction of the stored data. The actual saving depends on column widths, compression, metadata requests, and the storage system, but it does not require any special index: it follows directly from the physical separation of column chunks.\n\nPredicate pushdown uses information available near the storage layer to reduce the amount of data delivered to higher query operators. A query optimizer may pass a condition such as temperature > 30 to the Parquet reader. If a row group’s recorded maximum temperature is 18, that group cannot match and can be skipped. Minimum and maximum statistics are exclusion tools, not proof that a group contains matching rows. A group spanning temperatures from 10 to 40 still requires further processing to identify which rows satisfy the condition.\n\nOptional column and offset indexes extend this principle to pages. The column index describes page-level bounds and null information, while the offset index associates pages with their locations and row positions. When supported by the reader, these structures can avoid reading irrelevant pages and help identify corresponding row ranges in other columns. Their usefulness depends on data organization: randomly mixed values often produce broad bounds, whereas clustering records by time or another commonly filtered field creates tighter intervals.\n\nParquet can also store Bloom filters, which offer a different kind of exclusion test. A Bloom filter can establish that a searched value is absent from a column chunk, but a positive result means only that the value might be present. Such filters are especially useful for equality predicates when minimum and maximum bounds cover a large range. They are not substitutes for full indexes that identify exact matching row locations, and they introduce additional storage and writer-side computation.\n\nCorrect skipping requires careful treatment of logical ordering, nulls, floating-point special values, and historical statistics conventions. A reader must not interpret unsafe or unsupported statistics as a license to discard data. Likewise, a filter passed to a storage reader may still require evaluation after decoding; “pushdown” does not imply that every predicate can be resolved from metadata alone.\n\nAfter pages are read, many engines decode values into batches and execute operators over arrays rather than individual row objects. This approach can improve CPU-cache locality and reduce per-record overhead. Such vectorized execution belongs to the reader and execution engine, however; Parquet defines the stored representation, not the query-processing strategy."
    },
    {
      "heading": "Dataset layout, distributed storage, and updates",
      "text": "A single Parquet file is often only one component of a much larger dataset. Distributed systems commonly write many files so that workers can produce output independently and later scan it in parallel. Within each file, row groups provide further scheduling and skipping opportunities. The relationship between file size, row-group size, and page size is therefore important: each controls a different level of access granularity, metadata overhead, and parallel work.\n\nRow-group sizing involves trade-offs rather than a universal optimum. Larger row groups can improve compression and reduce the relative cost of metadata, but may require more writer memory and provide coarser statistics for pruning. Smaller row groups can improve selective access and create more processing units, while increasing overhead and sometimes weakening compression. Page size introduces similar tensions at a finer scale. Suitable choices depend on row width, nested structure, query selectivity, available memory, and the behavior of the particular implementation.\n\nDirectory partitioning is a separate dataset convention. A layout such as year=2025/month=03/ can let a query engine eliminate whole directories before opening files. The partition value may be inferred from the path and need not be physically repeated in every file. This mechanism should not be confused with row groups or Parquet’s internal statistics. A dataset can use partition pruning, file-level metadata, row-group statistics, and page indexes together, each eliminating work at a different stage.\n\nIn cloud computing, Parquet is frequently stored in object-storage services. Range requests allow readers to fetch the footer and selected data regions instead of downloading complete objects. Yet each request has latency and sometimes a direct monetary cost. An excessive number of tiny files can therefore be slower and more expensive to scan than a smaller number of well-sized files, even when the total data volume is identical. Compaction addresses this “small-files problem” by rewriting many small files into fewer larger ones.\n\nParquet does not define in-place record updates, dataset snapshots, or a protocol for atomic changes across files. In ordinary use, an update produces new files or replacement files rather than overwriting an individual compressed value. Appending rows to an existing file likewise requires a writer capable of updating its structure and footer; appending arbitrary record bytes is not sufficient.\n\nTable formats such as Apache Iceberg, Delta Lake, and Apache Hudi add metadata and commit protocols above data files. They can provide snapshots, schema management, and transactional behavior, although their mechanisms and supported file arrangements differ. These capabilities belong to the table layer, not to Parquet itself."
    },
    {
      "heading": "Implementations and relationships with other formats",
      "text": "Parquet’s specification is implemented by multiple libraries rather than by a single mandatory runtime. The Java implementation, commonly known as parquet-java, has been important in the Hadoop ecosystem. The C++ implementation within Apache Arrow supplies Parquet functionality to Arrow-based tools and to PyArrow in Python. Other systems maintain their own readers, writers, or execution-specific integrations. Agreement on the format permits data exchange, while differences in feature coverage make implementation details relevant.\n\nApache Arrow and Parquet solve related but distinct problems. Arrow defines columnar memory layouts intended for efficient computation and data interchange, while Parquet defines a persistent, encoded, and often compressed file representation. Reading Parquet into Arrow normally involves decoding and arranging values in Arrow buffers; Parquet should not generally be understood as a directly usable zero-copy memory image. Conversely, Arrow’s in-memory organization can make it a convenient target for batch-oriented Parquet readers.\n\nCompared with comma-separated values, Parquet carries an explicit schema, distinguishes binary data from text through types and annotations, and permits column-selective access. A CSV file is easy to inspect with basic text tools, but its consumers must agree on delimiters, quoting, character encoding, missing-value conventions, and type interpretation. JSON directly expresses nested structures and is useful for interchange and streaming records, but its textual field names and row-oriented organization generally make repeated analytical scans more expensive than a suitably organized columnar representation.\n\nApache Avro represents another design emphasis. Its common object-container format stores records together and provides explicit writer and reader schema resolution. This suits workflows centered on complete records and schema-governed message or data exchange. Parquet instead emphasizes reading selected columns from large datasets. Apache ORC is a closer comparison: it is also a columnar analytical format with compression, statistics, and indexing facilities, but uses different internal structures and encodings. Neither format is universally faster across all engines and workloads.\n\nInteroperability must be assessed at the feature level as well as the file-format level. A reader may support basic scalar columns yet lack support for a newer encoding, encryption mode, logical type, or nested-layout convention. Timestamp units and time-zone interpretation, decimal precision, unsigned integers, and older list representations are recurring points of difference. Compatibility tests often need to validate decoded values and schema meaning, not merely whether a file opens successfully.\n\nThe format’s widespread use also extends beyond SQL analytics. Machine-learning pipelines store feature tables, training examples, and intermediate results in Parquet. These applications benefit from selective loading and compact storage, although randomized access to individual examples may be less natural than sequential or batch-oriented scanning. The surrounding data loader determines how files and row groups are scheduled for model training."
    },
    {
      "heading": "Security, integrity, and implementation limits",
      "text": "Parquet’s modular encryption specification supports protecting file components rather than requiring encryption of a whole file as a single opaque object. It allows different columns to use different encryption keys and supports encrypted-footer and plaintext-footer arrangements. This can enable systems to expose selected metadata while restricting access to sensitive columns, or to protect the schema and other footer contents as well. The specification uses modes based on the Advanced Encryption Standard, including authenticated encryption.\n\nColumn encryption is not equivalent to a complete authorization system. An application still needs key management, identity handling, and access control that determine which users or services receive keys. External storage permissions and transport security remain relevant. Metadata design also matters: schemas, statistics, and auxiliary structures can reveal information even when direct access to column values is restricted. The chosen encryption configuration determines which components are protected, and support varies among implementations.\n\nIntegrity mechanisms operate at different levels. Parquet page headers can carry optional CRC checksums that help detect accidental corruption of page payloads. These checksums are not cryptographic authentication and do not protect against deliberate modification by an attacker who can recompute them. Authenticated encryption provides a different guarantee, binding protected content to authentication information. Even then, file-level protection does not by itself establish that an entire multi-file dataset is complete, current, or free from unauthorized deletion.\n\nReaders also process complex, externally supplied binary structures. They must validate lengths, offsets, nested levels, encoded counts, and decompression results. Malformed files can exercise bugs in parsers or compression libraries, while unusually large declared sizes or deeply nested structures can consume excessive resources. These are implementation-security concerns rather than properties unique to columnar storage. The format’s openness makes independent implementations possible, but it does not eliminate the need for defensive parsing and maintained dependencies.\n\nParquet has workload limits unrelated to security. It is not designed to replace an indexed transactional database for frequent single-record updates and low-latency point lookups. A selective query may still need to inspect metadata and decode pages containing many unrelated values. Very wide schemas can enlarge metadata and increase file-opening costs, while high-cardinality columns may obtain little benefit from dictionary encoding. Data whose values are already compressed or effectively random may compress poorly.\n\nPerformance measurements therefore depend on more than the filename extension. Codec choice, ordering, partition layout, row-group size, reader implementation, storage latency, and query shape all affect results. The same dataset can behave quite differently when scanned sequentially, filtered on a clustered timestamp, or searched for randomly distributed identifiers. These differences arise from the interaction between Parquet’s physical structures and the software using them."
    }
  ],
  "links": [
    "metadata",
    "data-warehouse",
    "data-lake",
    "database-management-system",
    "apache-hadoop",
    "apache-license",
    "open-source-software",
    "google",
    "file-system",
    "apache-spark",
    "interoperability",
    "utf-eight",
    "ieee-seven-five-four",
    "database-schema",
    "lossless-data-compression",
    "query-optimizer",
    "cpu-cache",
    "cloud-computing",
    "apache-iceberg",
    "database-transaction",
    "python-programming-language",
    "sql",
    "advanced-encryption-standard",
    "access-control"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}