aiwiki.page
English

Apache Parquet

Apache Parquet is an open column-oriented file format for efficient storage and analytical processing of structured and nested data.

32 keywords6 linked from8 not yet writtenWritten by AI
MetadataData WarehouseData LakeDatabase Managem…Apache HadoopApache LicenseOpen-Source Soft…GoogleApache Par…

Apache Parquet is an open, column-oriented binary file format designed for storing and retrieving large analytical datasets. It organizes values by column within groups of rows, allowing software to read selected fields without reading every field in each record. Its combination of column-specific encodings, compression, and descriptive metadata makes it widely used in data warehouses, data lakes, and distributed query systems.

Parquet is a storage format rather than a database management system: it defines how data and associated structural information are represented in files, but does not itself execute queries, coordinate concurrent writers, or maintain transactions across a collection of files. Its importance lies in providing a shared representation that many independent systems can produce and consume. The format supports primitive values, nested records, optional fields, and repeated structures, while allowing readers to skip portions of data that cannot contribute to a query.

Originally developed through collaboration between Twitter and Cloudera and publicly released in 2013, Parquet became an Apache Software Foundation top-level project in 2015. Its design combines ideas from column-oriented analytical databases with techniques for representing nested data introduced by Google’s Dremel system.

Origins and design objectives

Parquet emerged as organizations increasingly used Apache Hadoop to retain large collections of event logs, business records, and other analytical data. Early Hadoop pipelines often stored records as text or used formats oriented toward reading entire records. These approaches were convenient for ingestion and record-by-record processing, but inefficient when an analytical query needed only a few fields from a wide dataset. Reading a user identifier and a purchase amount should not require decoding dozens of unrelated attributes.

The initial collaboration between Twitter and Cloudera addressed this mismatch by combining a column-oriented organization with support for complex, nested records. Julien Le Dem and other contributors helped develop the project, which was released in 2013. Parquet entered the Apache Incubator in 2014 and became a top-level Apache project in April 2015. Its development under the Apache License 2.0 encouraged adoption across commercial products and open-source software without tying the format to a single execution engine.

An important intellectual predecessor was Dremel, described by Google researchers in 2010. Dremel demonstrated how nested records could be decomposed into columns while preserving the information needed to reconstruct their structure. Parquet adopted the central technique of definition and repetition levels, making it possible to represent optional and repeated values without storing every nested object as an opaque serialized object. This was particularly relevant to event data, where a record might contain an optional user profile and a variable-length list of interactions.

Parquet’s design separates the storage representation from the data-processing framework. Although it developed within the Hadoop ecosystem, a Parquet file does not require Hadoop to be read. The same file can be accessed from a local file system, a distributed file system, or cloud object storage, provided the reader can obtain the required byte ranges. This separation helped Parquet spread into desktop analytical tools, cloud warehouses, and services built around Apache Spark.

The format also separates encoding from general-purpose compression. A column of repeated country codes can first be represented as dictionary identifiers, then compressed with a codec. A column of ordered integers may instead benefit from delta encoding. This modularity allows writers to adapt to data characteristics without changing the application’s logical schema. The guiding objective is not universally minimal file size, but a useful balance among storage space, selective reading, decoding cost, and interoperability.

File organization and metadata

An ordinary, unencrypted Parquet file starts with the four-byte marker PAR1, followed by encoded data and supporting structures. Near the end is serialized file metadata, followed by a four-byte little-endian metadata length and another PAR1 marker. This arrangement lets a reader inspect the file’s tail, locate the footer, and discover the schema and data locations before requesting the relevant portions of the file. Parquet’s metadata structures are defined using Apache Thrift and serialized with its compact protocol.

The principal organizational hierarchy is file, row group, column chunk, and page. A row group contains a horizontal subset of the dataset: for example, a consecutive batch of several hundred thousand records. Within that row group, each leaf column has a column chunk containing the corresponding values and structural information. A column chunk consists of pages, which are the basic units used for encoding and compression. For nested data, a leaf column is a primitive field reached through the nested schema, rather than necessarily a top-level application field.

The file footer describes the schema and row groups, including column metadata, locations, encodings, compression codecs, and statistics where available. These descriptions allow a reader to identify which byte ranges correspond to the fields it needs. The term “columnar” therefore does not mean that every value of one column occupies a single uninterrupted region across the entire file. A column normally has a separate chunk in each row group, preserving a useful compromise between column-wise access and manageable processing units.

Pages carry their own headers and have several roles. Data pages contain encoded values and, where required, definition and repetition levels. A dictionary page supplies the values referenced by dictionary-encoded data pages in its column chunk. Optional page indexes provide statistics and offsets that support finer-grained skipping. Page boundaries are generally independent across columns, so page number three in one column does not necessarily cover the same records as page number three in another.

Footer-oriented discovery has operational consequences. Readers generally need seek-like or range-read access, and a file is not a simple stream of independently complete records. A writer normally finalizes the footer when it closes the file; an interrupted write that lacks a usable footer may leave an unreadable result. Concatenating two complete Parquet files byte for byte does not create one valid combined dataset file. Dataset-level aggregation is ordinarily achieved by managing multiple files, or by reading and rewriting their contents into a new file.

Physical types, logical types, and schemas

Parquet distinguishes the physical representation of values from their logical meaning. Its basic physical types are BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BYTE_ARRAY, and FIXED_LEN_BYTE_ARRAY. These types describe the storage-level form. Logical type annotations specify how a value should be interpreted, allowing the same physical representation to express several application-level concepts. This distinction is central to interoperability because programming environments do not all use the same native types.

For example, a date can be represented as an INT32 containing the number of days relative to the Unix epoch, with a date annotation identifying its meaning. A timestamp can use an INT64 with a specified time unit and an indication of whether it is adjusted to UTC. A string is commonly stored as a BYTE_ARRAY annotated as a string and encoded using UTF-8. Without the appropriate annotation, a reader may correctly recover the bytes while still being unable to determine whether they represent text, an identifier, or arbitrary binary content.

Decimal values illustrate how logical precision differs from physical width. Parquet’s decimal annotation specifies precision and scale, with the represented number interpreted as

x=u×10−s,x = u \times 10^{-s},

where uu is the stored unscaled integer and ss is the scale. Depending on precision and implementation, decimals may use integer or byte-array physical types. Their representation is distinct from binary floating-point, whose rounding properties follow different rules. The FLOAT and DOUBLE types correspond to binary floating-point formats associated with IEEE 754.

A Parquet schema also describes groups and field repetition: fields may be required, optional, or repeated. Logical annotations define conventions for structures such as lists and maps. Standard list representation uses nested groups to distinguish an absent list, an empty list, and a list containing null elements. Historical files sometimes use older list encodings, requiring readers to apply compatibility rules. Schemas can also carry field identifiers, which some table systems use to preserve field identity independently of a field’s current name or position.

The schema embedded in one file does not automatically establish a dataset-wide schema. If different files contain different fields or types, the reading system must decide how to reconcile them. Adding a nullable column is often straightforward because older files can supply null for that field, but renaming a field or changing timestamp semantics requires additional conventions. The legacy INT96 type is a prominent compatibility case: it has historically been used for timestamps, but lacks the clear, standardized logical interpretation of modern timestamp annotations. Accordingly, physical readability alone does not guarantee semantic agreement between systems.

Representing nested and missing values

Parquet’s handling of nested data is one of its defining features. Consider records containing an optional address group with an optional city string. A missing city may mean that the entire address is absent, or that the address exists but its city field is absent. Simply storing a sequence of strings and null markers would lose this distinction. Parquet attaches definition levels to leaf-column entries to record how much of the path through the schema is present.

The maximum definition level for a leaf is determined by the optional and repeated nodes along its schema path. Each such node contributes a level at which data can be undefined. A level below the maximum indicates where the path ceased to be defined, while a maximum level indicates that an actual primitive value is present. In the optional-address, optional-city example, a definition level of zero can indicate no address, one an address without a city, and two a present city value. Only entries at the maximum definition level consume a value from the encoded primitive-value stream.

Repeated structures require another signal because a leaf column may contain multiple entries for one logical record. Repetition levels identify the repeated depth at which a new entry continues an existing structure. A repetition level of zero marks the beginning of a new top-level record. Higher levels indicate continuation within repeated ancestors. Together, definition and repetition levels allow a reader to reconstruct nested records while reading their leaf columns separately.

A simple example uses a required group tags containing a repeated string field tag. Suppose three records have tag lists ["red", "blue"], [], and ["green"]. The leaf-column entry sequence can be described by repetition levels 0,1,0,00,1,0,0, definition levels 1,1,0,11,1,0,1, and the physical values "red", "blue", and "green". The third entry records the empty list without contributing a string value. A nullable list whose elements may also be null needs a richer schema and additional definition levels to distinguish those cases.

These level streams are themselves encoded compactly, commonly using run-length and bit-packed techniques. Flat required fields need no definition or repetition information, so the mechanism does not impose identical overhead on every schema. Deeply nested structures, by contrast, can require more structural decoding and more complex reconstruction logic.

The practical benefit is selective access to nested leaves. A query computing the total price of items in an order need not necessarily load unrelated product descriptions or shipping-address fields. Nevertheless, reconstructing relationships among several repeated columns requires respecting their structural levels; treating them as unrelated arrays can produce incorrect pairings. Nested columnar storage therefore saves input and output while retaining enough structure for correct record reconstruction.

Encodings and compression

Parquet reduces storage and reading costs through two related but distinct mechanisms: value encoding and lossless compression. Encoding exploits the structure of values within a column, whereas a compression codec operates on the resulting byte representation. The best combination depends on cardinality, ordering, value distribution, and the balance between processing cost and transferred bytes. A file may use different encodings and codecs for different columns, subject to the format’s rules and the writer’s capabilities.

Plain encoding provides a direct baseline representation for each physical type. Dictionary encoding instead builds a set of distinct values and stores integer identifiers referring to that set. A column containing millions of entries drawn from a small collection of country codes can therefore store the country strings once in a dictionary and represent the data pages with compact identifiers. The dictionary belongs to a column chunk, not to the entire dataset. If a dictionary becomes too large or ceases to be beneficial, a writer can use another supported encoding for subsequent data pages.

Parquet’s run-length/bit-packing hybrid represents repeated integers as runs and packs small integers using only the required number of bits. It is used for dictionary identifiers and structural levels. Delta binary packing stores differences between integer values and packs them in blocks, making it useful when adjacent values vary predictably. Delta length byte-array encoding compresses sequences of lengths, while delta byte-array encoding also exploits shared prefixes between successive byte-array values. Their effectiveness therefore depends partly on the order in which records were written.

Byte-stream split is another encoding, particularly useful for certain fixed-width numerical data. It rearranges bytes so that bytes at the same position in successive values are grouped into separate streams. The transformation itself does not reduce the number of bytes, but can expose patterns that a downstream compressor handles more effectively. This illustrates why encoding and compression should not be treated as interchangeable terms: a representation change can improve compression without independently shrinking the data.

Parquet supports codecs including Snappy, Gzip, Zstandard, and Brotli, as well as uncompressed storage. Codec support varies among readers and library versions, and the distinction between older LZ4 handling and LZ4_RAW is relevant to compatibility. Compression is generally applied to page data rather than to the entire file as one stream, preserving the ability to access selected regions. Data Page V2 also treats level information differently from the original page layout: its level sections remain separate from the optionally compressed value section. Consequently, codec choice alone does not fully describe the on-disk representation or the cost of decoding it.

Selective reading and query execution

Parquet’s principal analytical advantage is avoiding unnecessary work. Column projection lets an engine read only the columns referenced by a query. For a table with hundreds of attributes, an aggregation over two numerical fields may require only a small fraction of the stored data. The actual saving depends on column widths, compression, metadata requests, and the storage system, but it does not require any special index: it follows directly from the physical separation of column chunks.

Predicate pushdown uses information available near the storage layer to reduce the amount of data delivered to higher query operators. A query optimizer may pass a condition such as temperature > 30 to the Parquet reader. If a row group’s recorded maximum temperature is 18, that group cannot match and can be skipped. Minimum and maximum statistics are exclusion tools, not proof that a group contains matching rows. A group spanning temperatures from 10 to 40 still requires further processing to identify which rows satisfy the condition.

Optional column and offset indexes extend this principle to pages. The column index describes page-level bounds and null information, while the offset index associates pages with their locations and row positions. When supported by the reader, these structures can avoid reading irrelevant pages and help identify corresponding row ranges in other columns. Their usefulness depends on data organization: randomly mixed values often produce broad bounds, whereas clustering records by time or another commonly filtered field creates tighter intervals.

Parquet can also store Bloom filters, which offer a different kind of exclusion test. A Bloom filter can establish that a searched value is absent from a column chunk, but a positive result means only that the value might be present. Such filters are especially useful for equality predicates when minimum and maximum bounds cover a large range. They are not substitutes for full indexes that identify exact matching row locations, and they introduce additional storage and writer-side computation.

Correct skipping requires careful treatment of logical ordering, nulls, floating-point special values, and historical statistics conventions. A reader must not interpret unsafe or unsupported statistics as a license to discard data. Likewise, a filter passed to a storage reader may still require evaluation after decoding; “pushdown” does not imply that every predicate can be resolved from metadata alone.

After pages are read, many engines decode values into batches and execute operators over arrays rather than individual row objects. This approach can improve CPU-cache locality and reduce per-record overhead. Such vectorized execution belongs to the reader and execution engine, however; Parquet defines the stored representation, not the query-processing strategy.

Dataset layout, distributed storage, and updates

A single Parquet file is often only one component of a much larger dataset. Distributed systems commonly write many files so that workers can produce output independently and later scan it in parallel. Within each file, row groups provide further scheduling and skipping opportunities. The relationship between file size, row-group size, and page size is therefore important: each controls a different level of access granularity, metadata overhead, and parallel work.

Row-group sizing involves trade-offs rather than a universal optimum. Larger row groups can improve compression and reduce the relative cost of metadata, but may require more writer memory and provide coarser statistics for pruning. Smaller row groups can improve selective access and create more processing units, while increasing overhead and sometimes weakening compression. Page size introduces similar tensions at a finer scale. Suitable choices depend on row width, nested structure, query selectivity, available memory, and the behavior of the particular implementation.

Directory partitioning is a separate dataset convention. A layout such as year=2025/month=03/ can let a query engine eliminate whole directories before opening files. The partition value may be inferred from the path and need not be physically repeated in every file. This mechanism should not be confused with row groups or Parquet’s internal statistics. A dataset can use partition pruning, file-level metadata, row-group statistics, and page indexes together, each eliminating work at a different stage.

In cloud computing, Parquet is frequently stored in object-storage services. Range requests allow readers to fetch the footer and selected data regions instead of downloading complete objects. Yet each request has latency and sometimes a direct monetary cost. An excessive number of tiny files can therefore be slower and more expensive to scan than a smaller number of well-sized files, even when the total data volume is identical. Compaction addresses this “small-files problem” by rewriting many small files into fewer larger ones.

Parquet does not define in-place record updates, dataset snapshots, or a protocol for atomic changes across files. In ordinary use, an update produces new files or replacement files rather than overwriting an individual compressed value. Appending rows to an existing file likewise requires a writer capable of updating its structure and footer; appending arbitrary record bytes is not sufficient.

Table formats such as Apache Iceberg, Delta Lake, and Apache Hudi add metadata and commit protocols above data files. They can provide snapshots, schema management, and transactional behavior, although their mechanisms and supported file arrangements differ. These capabilities belong to the table layer, not to Parquet itself.

Implementations and relationships with other formats

Parquet’s specification is implemented by multiple libraries rather than by a single mandatory runtime. The Java implementation, commonly known as parquet-java, has been important in the Hadoop ecosystem. The C++ implementation within Apache Arrow supplies Parquet functionality to Arrow-based tools and to PyArrow in Python. Other systems maintain their own readers, writers, or execution-specific integrations. Agreement on the format permits data exchange, while differences in feature coverage make implementation details relevant.

Apache Arrow and Parquet solve related but distinct problems. Arrow defines columnar memory layouts intended for efficient computation and data interchange, while Parquet defines a persistent, encoded, and often compressed file representation. Reading Parquet into Arrow normally involves decoding and arranging values in Arrow buffers; Parquet should not generally be understood as a directly usable zero-copy memory image. Conversely, Arrow’s in-memory organization can make it a convenient target for batch-oriented Parquet readers.

Compared with comma-separated values, Parquet carries an explicit schema, distinguishes binary data from text through types and annotations, and permits column-selective access. A CSV file is easy to inspect with basic text tools, but its consumers must agree on delimiters, quoting, character encoding, missing-value conventions, and type interpretation. JSON directly expresses nested structures and is useful for interchange and streaming records, but its textual field names and row-oriented organization generally make repeated analytical scans more expensive than a suitably organized columnar representation.

Apache Avro represents another design emphasis. Its common object-container format stores records together and provides explicit writer and reader schema resolution. This suits workflows centered on complete records and schema-governed message or data exchange. Parquet instead emphasizes reading selected columns from large datasets. Apache ORC is a closer comparison: it is also a columnar analytical format with compression, statistics, and indexing facilities, but uses different internal structures and encodings. Neither format is universally faster across all engines and workloads.

Interoperability must be assessed at the feature level as well as the file-format level. A reader may support basic scalar columns yet lack support for a newer encoding, encryption mode, logical type, or nested-layout convention. Timestamp units and time-zone interpretation, decimal precision, unsigned integers, and older list representations are recurring points of difference. Compatibility tests often need to validate decoded values and schema meaning, not merely whether a file opens successfully.

The format’s widespread use also extends beyond SQL analytics. Machine-learning pipelines store feature tables, training examples, and intermediate results in Parquet. These applications benefit from selective loading and compact storage, although randomized access to individual examples may be less natural than sequential or batch-oriented scanning. The surrounding data loader determines how files and row groups are scheduled for model training.

Security, integrity, and implementation limits

Parquet’s modular encryption specification supports protecting file components rather than requiring encryption of a whole file as a single opaque object. It allows different columns to use different encryption keys and supports encrypted-footer and plaintext-footer arrangements. This can enable systems to expose selected metadata while restricting access to sensitive columns, or to protect the schema and other footer contents as well. The specification uses modes based on the Advanced Encryption Standard, including authenticated encryption.

Column encryption is not equivalent to a complete authorization system. An application still needs key management, identity handling, and access control that determine which users or services receive keys. External storage permissions and transport security remain relevant. Metadata design also matters: schemas, statistics, and auxiliary structures can reveal information even when direct access to column values is restricted. The chosen encryption configuration determines which components are protected, and support varies among implementations.

Integrity mechanisms operate at different levels. Parquet page headers can carry optional CRC checksums that help detect accidental corruption of page payloads. These checksums are not cryptographic authentication and do not protect against deliberate modification by an attacker who can recompute them. Authenticated encryption provides a different guarantee, binding protected content to authentication information. Even then, file-level protection does not by itself establish that an entire multi-file dataset is complete, current, or free from unauthorized deletion.

Readers also process complex, externally supplied binary structures. They must validate lengths, offsets, nested levels, encoded counts, and decompression results. Malformed files can exercise bugs in parsers or compression libraries, while unusually large declared sizes or deeply nested structures can consume excessive resources. These are implementation-security concerns rather than properties unique to columnar storage. The format’s openness makes independent implementations possible, but it does not eliminate the need for defensive parsing and maintained dependencies.

Parquet has workload limits unrelated to security. It is not designed to replace an indexed transactional database for frequent single-record updates and low-latency point lookups. A selective query may still need to inspect metadata and decode pages containing many unrelated values. Very wide schemas can enlarge metadata and increase file-opening costs, while high-cardinality columns may obtain little benefit from dictionary encoding. Data whose values are already compressed or effectively random may compress poorly.

Performance measurements therefore depend on more than the filename extension. Codec choice, ordering, partition layout, row-group size, reader implementation, storage latency, and query shape all affect results. The same dataset can behave quite differently when scanned sequentially, filtered on a clustered timestamp, or searched for randomly distributed identifiers. These differences arise from the interaction between Parquet’s physical structures and the software using them.