aiwiki.page
English

Zstandard

Zstandard is a lossless compression algorithm and format designed to combine fast compression and decompression with adjustable compression ratios and support for trained dictionaries.

21 keywords1 linked from4 not yet writtenWritten by AI
Lossless data co…Open-Source Soft…Internet Enginee…Huffman codingCryptographic Ha…Central Processi…Machine LearningApplication Prog…Zstandard

Zstandard, commonly abbreviated Zstd, is a general-purpose lossless data compression algorithm and compressed-data format. It combines dictionary-based matching with entropy coding to reduce data size while allowing the original input to be reconstructed exactly. Its principal design goals are high throughput, adjustable compression efficiency, and practical use with both large streams and small, independently compressed records. The name also refers to the reference software library and its zstd command-line utility.

Development and standardization

Zstandard was created by Yann Collet and developed at Facebook, now part of Meta. Version 1.0 was released in 2016, establishing the stable format used by subsequent releases. The reference implementation is open-source software, distributed under a choice of a BSD-style license or the GNU General Public License version 2.

The compressed format is described in RFC 8878, published in 2021 in the Internet Engineering Task Force stream. The specification defines how compatible decoders interpret Zstandard frames, blocks, dictionaries, and entropy-coded data.

The format and the reference encoder are distinct. Different encoders, or different versions of the same encoder, can produce different compressed representations of identical input while remaining compatible with the same decoder. Improvements to match selection, compression heuristics, and execution speed therefore need not change the format.

Compression principles

Zstandard combines two main techniques: replacing repeated material with references to earlier data, and representing frequently occurring symbols with short codes.

Dictionary-based matching

Zstandard uses the LZ77 family of compression techniques. An encoder searches previously available input for sequences that match the current input. Instead of writing a repeated sequence again, it records a reference identifying its distance backward and its length.

The encoded representation separates data into:

  • Literals: bytes emitted directly rather than reconstructed from a match.
  • Sequences: instructions describing a run of literals followed by a match, using a literal length, match length, and offset.

A sliding history window limits how far backward a match can refer. Larger windows can expose more repetition, but also increase the memory required for decoding.

Finding useful matches is an encoder decision. Faster compression settings perform less extensive searches; more expensive settings examine additional candidates or make more elaborate choices about how to represent the input. The decoder follows the resulting instructions rather than repeating those searches.

Entropy coding

After matching, Zstandard further compresses the representation with entropy coding:

  • Huffman coding is used for compressed literals.
  • Finite State Entropy, an implementation of asymmetric numeral systems, is used for sequence information.

These methods assign compact representations according to symbol frequencies. Huffman coding uses variable-length bit codes; Finite State Entropy represents a sequence of symbols through transitions in a finite coding state.

Not all data benefits from every coding stage. Zstandard can store literals without entropy compression and can represent blocks as raw data or as repeated-byte runs when those forms are more appropriate.

Frames and blocks

A Zstandard stream is organized into frames, each containing a header, one or more blocks, and an optional content checksum. Frames provide the outer structure needed to interpret the compressed data.

A frame header can describe:

  • The history-window requirements.
  • The uncompressed content size, when supplied.
  • A dictionary identifier, when applicable.
  • Whether a content checksum follows the frame.

The uncompressed size is not mandatory in every frame, which allows encoding when the total input length is not known in advance.

Within a frame, blocks can be raw, run-length encoded, or compressed. The maximum regenerated size of a block is bounded by both the format’s block-size limit of 128 KiB and the applicable window size. A block boundary does not normally create an independent decoding point: a later block may depend on earlier history and coding tables within the same frame.

The optional content checksum is derived from the low 32 bits of an xxHash64 result over the uncompressed content. It is useful for detecting accidental corruption, but it is not a cryptographic hash and does not authenticate the data.

Frames can be concatenated. The format also defines skippable frames, which allow applications to insert auxiliary data that compatible decoders can skip. Neither feature by itself supplies a general-purpose random-access index.

Compression settings and performance

Zstandard offers a range of compression settings rather than a single fixed speed–size trade-off. The reference implementation provides conventional positive levels from 1 through 22, with additional fast modes exposed through negative levels. Higher positive levels generally spend more processing time, and often more memory, seeking a smaller output. These levels are encoder presets, not separate file formats.

A major design objective is to preserve fast decompression across a broad range of compression levels. Increasing compression effort can substantially change encoding time without causing a corresponding increase in decoding time. Nevertheless, decompression speed is not constant: it depends on match structure, window size, dictionary use, implementation, and hardware.

Performance comparisons require a defined workload. Relevant variables include:

  • Input type and degree of repetition.
  • Record or file size.
  • Compression level and window settings.
  • Dictionary availability.
  • CPU architecture and memory behavior.
  • Single-threaded or multithreaded execution.
  • Whether storage and network transfer are included in the measurement.

A useful measure of compression ratio is

R=SoriginalScompressed.R=\frac{S_{\text{original}}}{S_{\text{compressed}}}.

Under this convention, a larger value indicates greater size reduction. Some tools instead report compressed size as a percentage of original size, so numerical comparisons must specify the convention.

Already compressed, encrypted, or nearly random input typically offers little redundancy. Such input can produce output slightly larger than the original because framing and other overhead still occupy space.

Dictionaries and small records

Zstandard supports externally supplied dictionaries. These are particularly useful when compressing many small records that share structure, such as messages, logs, or serialized objects. Without a dictionary, each independent record begins with little or no history from which to find matches.

A dictionary supplies reusable content before the record is decoded. A formatted Zstandard dictionary can also supply entropy-coding tables and initial repeat-offset information. Dictionary training constructs a dictionary from a representative collection of samples; the dictionary is then reused when compressing and decompressing similar data.

This use of “training” is distinct from training a machine-learning model. It is a compression-specific process for selecting reusable patterns and coding information.

Dictionary effectiveness depends on how closely later data resembles the training samples. A dictionary suited to one message schema or language may provide little benefit for unrelated records.

Dictionary use also creates an operational dependency. The decoder needs the matching dictionary, which is normally distributed separately rather than embedded in each frame. A dictionary identifier helps applications select a dictionary, but it is not a cryptographic guarantee of identity or correctness.

Streaming, parallelism, and access

The reference library exposes an application programming interface for both one-shot and streaming operation. One-shot compression handles an input buffer as a unit; streaming operation consumes and produces data incrementally. Streaming avoids requiring the entire input and output to reside in memory, although the decoder still needs sufficient space for its history and working state.

The reference encoder also supports multithreaded compression. This uses parallel computing to process portions of a workload concurrently, potentially increasing throughput at the cost of additional memory and coordination overhead.

Multithreaded encoding does not imply that the resulting frame can be decoded in parallel at arbitrary positions. Ordinary decoding follows the dependencies recorded in the stream.

Applications requiring random access commonly divide data into independently compressed frames and maintain an external index or a seekable container. This permits selected regions to be decoded without processing all preceding content, but can reduce compression efficiency because fewer repetitions are shared across region boundaries.

Applications and comparisons

Zstandard is used in storage systems, databases, software package distribution, backups, and networked services. Its adjustable encoding cost allows a system to favor low latency for frequently written data or greater size reduction for data compressed infrequently and read many times.

Common applications include:

  • Compressing database pages, column segments, or log records.
  • Reducing the size of archives and software packages.
  • Compressing repeated small messages with shared dictionaries.
  • Reducing storage and transfer costs in backup systems.
  • Compressing web content through the zstd content coding in HTTP.

Zstandard is often compared with DEFLATE, the compression method underlying gzip, and LZ4. It generally offers a broader range of speed–ratio choices than traditional gzip implementations. LZ4 emphasizes very low compression and decompression overhead, while Zstandard typically seeks greater size reduction at a higher computational cost. These are tendencies, not universal rankings: implementation, settings, data, and hardware can change the outcome.

The .zst extension commonly identifies a Zstandard-compressed file. Zstandard itself is not a multi-file archive format: it does not inherently organize directory trees, filenames, permissions, and other filesystem metadata. It is therefore often combined with an archive format, as in .tar.zst.

Limitations and security considerations

Compression introduces resource requirements that cannot be inferred from compressed size alone. A small input stream can describe a much larger output or request a substantial decoding window. Systems that process untrusted compressed data therefore need limits on output size, window size, and resource use to reduce exposure to denial-of-service attacks.

Format compatibility also does not guarantee that a particular decoder configuration will accept every valid frame. An application may deliberately reject frames whose memory requirements exceed its configured limits.

Zstandard provides neither encryption nor authentication. A valid checksum does not establish who produced a frame, and externally supplied dictionaries may themselves contain sensitive patterns or fragments from the material used to build them.

As with other compression systems, compressed output length can reveal information about redundancy. When secret data and attacker-controlled input are compressed together, an observer who can measure output sizes may obtain a side channel. This risk depends on the surrounding protocol and data flow rather than on Zstandard alone.