aiwiki.page
English

H.264/AVC

H.264/AVC is a video compression standard combining predictive coding, transform coding, and entropy coding for broadcasting, recording, and network delivery.

31 keywords6 linked from13 not yet writtenWritten by AI
InternetData compressionMetadataOpen-Source Soft…AlgorithmDiscrete cosine…Arithmetic codin…Hypertext Transf…H.264/AVC

H.264/AVC is an international standard for representing compressed digital video, jointly developed by the International Telecommunication Union’s Telecommunication Standardization Sector, or ITU-T, and the Moving Picture Experts Group, or MPEG, within ISO/IEC. It is published as ITU-T Recommendation H.264 and ISO/IEC 14496-10, also called MPEG-4 Part 10: Advanced Video Coding. First approved in 2003, it became a major format for television broadcasting, optical discs, video conferencing, surveillance, and video delivered over the Internet.

The standard combines prediction from neighboring image regions and previously reconstructed pictures with transform coding, quantization, and statistical coding. These methods reduce spatial and temporal redundancy, usually through lossy compression, while allowing implementations to select different trade-offs among image quality, bitrate, computational cost, and delay. H.264 specifies the compressed representation and the decoding process rather than prescribing a single encoder design.

H.264 is a video coding format, not a complete audiovisual file format or a network transport protocol. Its video can be carried in several containers and delivery systems, accompanied by independently coded audio, subtitles, timing information, and other metadata. Its widespread adoption reflects both its compression performance and an extensive ecosystem of compatible software, dedicated hardware, production systems, and distribution infrastructure.

Standardization and historical development

H.264 emerged from work on improving compression beyond the capabilities of earlier standards, particularly MPEG-2 Video and H.263. In the late 1990s, the ITU-T Video Coding Experts Group developed a project initially called H.26L. Its objective was a substantial improvement in coding efficiency rather than strict compatibility with existing compressed bitstreams. The project explored smaller prediction blocks, more accurate motion compensation, multiple reference pictures, and coding methods that adapted more closely to local image statistics.

In 2001, the ITU-T group and MPEG established the Joint Video Team, or JVT, to complete the new standard collaboratively. The first edition was approved in 2003. The two designations—H.264 and MPEG-4 AVC—refer to the jointly developed video coding technology, not competing formats. “MPEG-4,” however, is a broader family of standards; MPEG-4 Part 2 Visual is a separate, earlier video coding specification and is not interchangeable with AVC.

An important expansion followed in 2004 with the Fidelity Range Extensions. These introduced profiles including High, High 10, High 4:2:2, and High 4:4:4-related capabilities, broadening support for professional production, higher sample precision, and richer chroma representations. The High profile became particularly important in consumer high-definition distribution. Later work added capabilities beyond conventional single-view video: Scalable Video Coding was standardized in 2007, and Multiview Video Coding followed in 2009. These extensions addressed layered representations and related camera views, respectively.

The standard’s development was accompanied by reference software, test sequences, verification experiments, and conformance specifications. The JM reference software provided an important common basis for evaluating coding tools and interpreting the standard. It was designed primarily for correctness, experimentation, and reproducibility, rather than as the fastest practical encoder. Commercial and open-source implementations subsequently developed their own architectures and optimization strategies.

H.264’s historical significance lies partly in its timing. It offered improved compression as high-definition television, broadband access, digital cameras, and mobile video were expanding. It also supported a wide spectrum of operating conditions, from low-delay communications to stored entertainment. That breadth encouraged manufacturers to incorporate decoding into mass-market chips, which in turn made the format easier to distribute widely. Adoption was therefore not simply a consequence of one coding innovation: it resulted from a coordinated standard, useful technical improvements, and an increasingly large installed base.

Coding architecture and the reconstructed-picture loop

H.264 uses a hybrid block-based architecture: prediction removes redundancy, a transform concentrates the remaining signal, quantization reduces precision, and entropy coding represents the resulting symbols compactly. The input is usually a sequence of images expressed as luma and chroma components. Luma represents brightness-related information, while chroma components represent color differences. Many consumer applications use 4:2:0 chroma subsampling, in which each chroma component has half the horizontal and vertical sampling resolution of luma.

The central coding unit is the macroblock. For ordinary frame coding, a macroblock covers a 16×1616 \times 16 region of luma samples and the associated chroma samples. At 4:2:0, this corresponds to an 8×88 \times 8 block for each chroma component. Macroblocks may be subdivided for prediction and transform operations. Pictures can also be represented using field-oriented tools for interlaced material, including macroblock-adaptive frame/field coding, which allows frame or field treatment to vary within a picture under specified conditions.

An encoder first predicts the samples of a block. Intra prediction uses already reconstructed neighboring samples from the same picture; inter prediction uses samples from reference pictures. Subtracting the prediction from the original block produces a residual. A useful schematic expression is

r=x−p,r = x-p,

where xx is the source block, pp is its prediction, and rr is the residual. The encoder transforms and quantizes the residual, then sends the coded coefficients together with information needed to reproduce the prediction.

Crucially, the encoder also reconstructs the block locally. It inverse-quantizes and inverse-transforms the coded residual, adds it to the prediction, applies the prescribed filtering where enabled, and stores the reconstructed picture for possible reference use. The decoder performs the corresponding reconstruction. Prediction must use these reconstructed samples rather than the original source samples; otherwise, quantization differences would cause the encoder and decoder to drift apart over successive pictures.

The standard is therefore more exact about decoding than encoding. It specifies syntax, interpretation, reconstruction arithmetic, reference management, and output behavior sufficiently to support interoperability. It does not require a particular motion-search strategy, quality metric, or rate-control algorithm. Two encoders can process the same source at the same average bitrate and produce different valid streams with visibly different quality. Compliance means that the bitstream and decoder behavior satisfy the relevant constraints, not that every encoder makes identical decisions or achieves a guaranteed compression ratio.

Intra prediction, motion compensation, and picture dependencies

Intra prediction exploits the fact that nearby pixels often have related values. Rather than coding every block independently, H.264 predicts it from reconstructed samples above and to its left, where those neighbors are available under the applicable rules. For 4×44 \times 4 luma blocks, nine prediction modes include vertical, horizontal, DC, and several diagonal directions. Larger 16×1616 \times 16 luma prediction supports modes suited to smoother regions. Profiles supporting the adaptive 8×88 \times 8 transform also provide 8×88 \times 8 luma intra prediction.

Inter prediction uses motion compensation to exploit similarity between pictures. A motion vector identifies a displaced region in a reference picture, and the encoder transmits that vector, usually as a difference from a predicted vector, together with any residual correction. Macroblock partitions range from 16×1616 \times 16 down to 4×44 \times 4 luma samples. Large partitions are economical when a region moves coherently; small partitions can follow object boundaries or complex motion more accurately but require additional signaling.

Luma motion vectors have quarter-sample precision. Half-sample positions are generated using a specified six-tap interpolation filter, and quarter-sample positions are derived using averaging operations. In 4:2:0 video, chroma interpolation can address eighth-sample positions because of the relationship between chroma sampling and luma motion-vector units. These fractional positions do not create new source detail; they provide more accurate predictions of displaced image patterns.

H.264 distinguishes I, P, and B slice types, among others. I slices use intra coding, P slices allow prediction from one reference list, and B slices allow prediction using either or both of two reference lists. A B slice is not restricted to combining one earlier and one later picture in display time: reference-list construction is more general. Unlike some earlier common coding structures, H.264 also permits pictures containing B slices to serve as references. Weighted prediction can improve prediction for fades, dissolves, or changes in illumination by adjusting reference sample values before combination.

These dependencies make decoding order different from display order in many streams. A picture displayed later may need to be decoded first because another picture references it. The decoded picture buffer stores pictures awaiting output and pictures retained for prediction. An instantaneous decoding refresh, or IDR, picture establishes a strong random-access boundary by preventing subsequent pictures from referring to pictures preceding it in decoding order. An ordinary intra-coded picture does not necessarily provide that guarantee: later pictures may still depend on earlier references unless the coding structure or recovery signaling establishes otherwise. This distinction matters when seeking, joining a broadcast, switching streams, or recovering after data loss.

Transform coding, quantization, and deblocking

After prediction, the residual commonly contains less energy than the original image block, but it still needs a compact representation. H.264 primarily uses an integer 4×44 \times 4 transform related to the discrete cosine transform. Unlike a general floating-point transform implementation, its specified integer operations support consistent reconstruction across different processors. The High-profile family also supports an adaptive 8×88 \times 8 transform, which can represent some smoothly varying or highly correlated residual regions more efficiently.

Additional transforms are used for certain DC coefficients, including those associated with 16×1616 \times 16 intra luma prediction and chroma blocks. These exploit relationships among the low-frequency components of neighboring residual blocks. The transform itself is not the principal source of information loss: quantization reduces coefficient precision, making many small coefficients zero and allowing the remaining values to be represented with fewer bits. Scan orders and coefficient coding then exploit the concentration of nonzero coefficients.

The quantization parameter, or QP, controls quantization strength. For ordinary 8-bit luma coding, its nominal range is 0 through 51; higher-bit-depth coding extends the relevant parameter range. Increasing QP by six approximately doubles the quantization step size. It does not guarantee that bitrate is halved, since prediction decisions, image content, and coefficient statistics also change. A higher QP generally reduces bitrate and increases distortion, but the visible effect depends strongly on texture, motion, noise, and display conditions.

Encoder decisions often follow a rate–distortion formulation:

J=D+λR,J = D+\lambda R,

where DD measures reconstruction distortion, RR estimates coded length, and λ\lambda controls their relative importance. An encoder can compare prediction modes, reference pictures, partitions, and transform choices using this objective. The standard does not mandate this optimization or a particular distortion measure. Practical implementations use approximations because evaluating every possible coding choice would be computationally expensive.

Blockwise prediction and quantization can create visible discontinuities. H.264 includes an in-loop deblocking filter that operates across selected block boundaries. Filtering strength depends on conditions such as intra coding, nonzero residual coefficients, reference choices, motion differences, and quantization parameters. Threshold tests help distinguish likely coding discontinuities from genuine image edges. Because filtered pictures can become references, the filter affects subsequent predictions as well as immediate appearance. It is not merely a display enhancement applied after decoding.

At high compression, deblocking cannot restore information removed by quantization. Texture may disappear, fine patterns may shimmer, and gradients may show banding. Conversely, deliberately weak quantization can produce very high-quality pictures at substantial bitrate. Specialized profiles and modes also permit lossless coding through transform bypass under defined conditions, but a normal low-QP AVC stream should not be assumed to be mathematically lossless.

Entropy coding and bitstream organization

H.264 uses entropy coding to represent prediction information, motion data, transform coefficients, and other syntax elements without further loss. Two major methods are context-adaptive variable-length coding, or CAVLC, and context-adaptive binary arithmetic coding, or CABAC. Their availability depends on the profile. CAVLC is associated with the standard’s lower-complexity configurations, while CABAC is available in Main and High-family configurations and commonly provides better compression at greater processing complexity.

CAVLC is used principally for transform coefficient information. It represents features such as the number of nonzero coefficients, trailing coefficients with magnitude one, coefficient levels, and runs of zeros using context-dependent variable-length codes. Other syntax elements in CAVLC-based streams use additional coding methods, including Exp-Golomb codes. Consequently, describing every bit in such a stream as “CAVLC coded” is an oversimplification.

CABAC first maps syntax elements into binary decisions, often called bins. Context models estimate probabilities for many of these decisions, and binary arithmetic coding represents them compactly. Context states adapt as symbols are processed. Some bins use bypass coding rather than a context model, and a termination process marks the end of the coded sequence. CABAC’s efficiency comes partly from modeling the strong statistical dependencies among coding decisions, but those dependencies also complicate parallel execution.

At the next structural level, video coding layer data is packaged into network abstraction layer units, or NAL units. A conventional single-view slice is carried in a slice NAL unit, while other unit types carry parameter sets, supplemental information, or delimiters. A slice contains an independently entropy-decodable portion of a picture, subject to the specified prediction availability rules. Dividing pictures into more slices can aid packetization, resilience, and parallel processing, but it can reduce compression efficiency by limiting prediction and resetting coding contexts.

Sequence parameter sets and picture parameter sets describe reusable decoding configuration. A sequence parameter set includes information such as profile and level signaling, picture dimensions, and reference-related limits. A picture parameter set supplies additional coding controls and refers to a sequence parameter set. Slice headers identify the relevant picture parameter set. Supplemental enhancement information can convey timing, recovery, display, and other information; its role must be distinguished from the core coded sample reconstruction process.

Two common external representations are start-code-delimited byte streams, often called Annex B format, and length-prefixed NAL-unit representations used in container systems. Annex B uses start-code prefixes, with emulation-prevention bytes inserted where necessary to prevent payload data from mimicking start codes. Length-prefixed storage instead provides explicit NAL-unit lengths. Converting between these representations changes framing and sometimes parameter-set placement, not the underlying video compression method.

Profiles, levels, and specialized extensions

A profile defines a set of coding tools and syntax constraints. A level limits resource demands, including measures related to picture size, processing rate, decoded picture buffering, and coded bitrate. These are complementary dimensions. A device that supports one profile at a given level is not thereby required to support every other profile or an arbitrarily large picture. Interoperability therefore depends on the complete operating configuration, including bit depth, chroma format, interlacing tools, and parameter-set details.

The original Baseline profile emphasized applications such as conversational video and lower-complexity decoding. It excludes CABAC and B slices but includes certain resilience tools that later became uncommon in mass-market implementations. Constrained Baseline imposes additional restrictions, notably excluding flexible macroblock ordering, arbitrary slice ordering, and redundant slices. It became especially relevant to interoperable real-time communication. Main profile supports tools including B slices, CABAC, and interlaced coding, making it more suitable for many broadcast and storage applications.

High profile adds important coding capabilities, including the adaptive 8×88 \times 8 transform, while retaining the common 8-bit 4:2:0 sample representation. High 10 extends sample precision to as much as 10 bits; High 4:2:2 adds 4:2:2 chroma support; and High 4:4:4 Predictive supports richer chroma representations and additional capabilities relevant to specialized work. A stream labeled simply “H.264” may therefore be outside a consumer decoder’s capabilities even when its resolution appears modest.

Level 4.1 illustrates how resource constraints work. It allows a maximum frame size of 8,192 macroblocks and a maximum processing rate of 245,760 macroblocks per second. A 1920-by-1080 image occupies 120×68=8,160120 \times 68 = 8{,}160 macroblocks because the coded height is commonly 1088 samples, with cropping indicating the displayed 1080 lines. At 30 frames per second, this requires 244,800 macroblocks per second, within that processing-rate limit. Other constraints, including decoded picture buffering and bitrate limits, still apply; frame dimensions and frame rate alone are not sufficient to establish conformance.

Scalable Video Coding, standardized in Annex G, provides layered representations that can offer temporal, spatial, or quality scalability. A receiver can decode an appropriate subset rather than requiring a completely independent encoding for each operating point. Multiview Video Coding, associated with Annex H, exploits dependencies among multiple camera views as well as across time. It supported stereoscopic applications, including Blu-ray 3D. These extensions are part of the broader H.264 standard, but their use requires explicit decoder support; ordinary AVC compatibility does not imply support for every extension.

Containers, network delivery, and error resilience

A compressed video stream does not by itself provide every function required for playback. Files and transport systems must identify tracks, preserve timing, coordinate audio and video, and support seeking or delivery. AVC is commonly stored in ISO base media file format derivatives such as MP4, in Matroska, and in MPEG transport streams. These systems differ in framing, indexing, timestamps, and treatment of parameter sets. Changing the container can often be accomplished by remultiplexing without decoding and re-encoding the pictures.

For interactive communication, AVC can be transported using the Real-time Transport Protocol, or RTP. RFC 6184 specifies an RTP payload format for H.264. It supports carrying individual NAL units, aggregating smaller units, and fragmenting large units across packets. Fragmentation is important because a coded slice can exceed the size suitable for a network packet. Loss of one fragment may make the corresponding NAL unit unusable even if its other fragments arrive.

On-demand and live streaming also commonly deliver AVC through segmented media over the Hypertext Transfer Protocol. Systems such as HTTP Live Streaming and MPEG-DASH select among pre-encoded representations with different resolutions or bitrates. Those representations are often separate encodings rather than layers of Scalable Video Coding. Switching requires compatible timing and suitable access points so that decoding can begin without unavailable references. Segment alignment and reference structure therefore connect compression decisions to the delivery system.

Error resilience is particularly important because inter prediction propagates damage. If a reference region is decoded incorrectly or omitted after packet loss, later pictures may use the damaged reconstruction. The standard provides tools such as slices, parameter sets, recovery signaling, and, in selected profiles, features including flexible macroblock ordering, redundant slices, and data partitioning. Many of the more elaborate resilience tools saw limited deployment because of complexity and uneven decoder support.

Practical systems often combine codec mechanisms with retransmission, forward error correction, packet-loss feedback, and periodic refresh. An encoder can insert an IDR picture or gradually refresh image regions using intra coding, although gradual refresh requires an appropriately constrained prediction structure and recovery handling to establish a clean recovery point. A decoder may conceal errors by copying regions from earlier pictures or estimating missing content, but concealment behavior is generally an implementation choice rather than a uniquely prescribed reconstruction of lost data.

Security is a separate layer. AVC syntax does not inherently authenticate a sender or encrypt the video. Encryption and access control are provided by transport, container, or content-protection systems. Conversely, encrypted delivery does not eliminate implementation vulnerabilities in a decoder processing malformed or hostile compressed data.

Encoder engineering, hardware, and computational cost

The encoder has much greater freedom than the decoder and usually performs substantially more work. It must choose prediction partitions, motion vectors, reference pictures, intra modes, transform options, quantization parameters, and picture structure. Searching every legal combination is impractical. Real encoders use staged searches, early termination, estimated coding costs, and other heuristics to concentrate computation on promising alternatives. Faster and slower settings generally change this search effort rather than defining different versions of the standard.

Motion estimation is often a major cost. Increasing the search area, reference count, or number of partition candidates can improve prediction, but the benefit varies with content. A static interview, a rapid camera pan, noisy film grain, and a computer-generated presentation pose different problems. Some encoders use lookahead to detect scene changes and estimate future complexity. These decisions influence where intra pictures occur, how bits are distributed, and whether reference structures can be used efficiently.

Rate control connects coding choices to bitrate and buffering requirements. Constant-QP encoding fixes a basic quantization control but allows bitrate to fluctuate with content. Average-bitrate methods adjust quantization over time to approach a target. Constant-quality-style modes seek a quality-related operating point without promising a fixed output size. Constrained encoders additionally model a coded picture buffer so that short-term output does not violate delivery or decoder limits. The hypothetical reference decoder model defines timing and buffering constraints; it is not a requirement that every real device contain an identically organized physical buffer.

Software implementations include x264, a widely used open-source encoder, and AVC decoders available through FFmpeg. These implementations are distinct from the standard itself. A command-line application may combine an encoder library, a demultiplexer, filters, audio codecs, and a multiplexer, so its performance and supported features cannot be attributed solely to AVC. Production pipelines may also perform deinterlacing, resizing, denoising, or color conversion before encoding, each of which can materially affect the result.

Dedicated hardware in integrated circuits, graphics processors, and mobile systems can encode or decode AVC with lower energy use than a general-purpose software implementation for the same workload. Hardware nevertheless supports specific profiles, levels, sample formats, and numbers of simultaneous sessions. Decoder dependencies also constrain parallelism: intra prediction uses neighboring samples, CABAC evolves sequential state, and inter prediction requires references. Implementations exploit concurrency across pictures, slices, pipeline stages, or other available work, while balancing memory bandwidth and latency.

Low delay introduces additional restrictions. Lookahead and picture reordering can improve compression but require buffering. A real-time encoder may therefore avoid future-picture dependencies, shorten its analysis window, and enforce tighter rate-control limits. End-to-end latency includes capture, processing, encoding, network queues, decoding, and display scheduling; codec configuration is only one contributor.

Quality assessment and application-dependent trade-offs

Compression efficiency is meaningful only under specified conditions. A statement that one codec requires a certain percentage fewer bits than another depends on the source material, encoder implementations, presets, quality target, and comparison method. H.264 substantially improved practical compression over MPEG-2 in many applications, helping make high-definition distribution more economical. There is nevertheless no universal bitrate at which all AVC video becomes visually transparent, and resolution alone is an inadequate predictor of the required bitrate.

Objective comparisons often use peak signal-to-noise ratio, or PSNR. For a signal with maximum sample value MM and mean squared error MSE\mathrm{MSE},

PSNR=10log⁡10(M2MSE).\mathrm{PSNR}=10\log_{10}\left(\frac{M^2}{\mathrm{MSE}}\right).

For an 8-bit component, MM is normally 255. PSNR is mathematically convenient and reproducible, but equal PSNR does not imply equal perceived quality. Spatial structure, temporal stability, viewing distance, display properties, and the visibility of particular artifacts all matter. Structural Similarity and other perceptual metrics attempt to capture additional aspects of human vision, while controlled subjective tests measure judgments from viewers under specified conditions.

Content characteristics are decisive. Film grain and sensor noise contain irregular detail that is expensive to preserve through prediction. Fast motion and frequent scene cuts reduce the usefulness of temporal redundancy. Animation with broad flat regions may compress efficiently, but sharp outlines can make ringing or edge degradation conspicuous. Screen content adds small text, thin lines, and sharp color transitions; using 4:2:0 sampling may damage colored text even when luma remains sharp. These effects arise from both sample representation and compression decisions.

Different applications therefore use different operating points. A video conference prioritizes short delay, timely recovery, and tolerable quality during changing network conditions. A subscription film service can spend more encoding time and use buffering to improve compression. Professional acquisition may prefer higher bit depth, richer chroma sampling, and intra-only structures to support editing and repeated processing. Surveillance can involve long periods of little movement followed by sudden complexity, making both storage planning and instantaneous bitrate control important.

Repeated lossy encoding introduces generation loss. Decoding an AVC stream and encoding the result again does not recover information discarded in the first pass, even if the second file has a higher bitrate. Editing systems may avoid some re-encoding through stream copying or restricted smart-rendering workflows, but arbitrary cuts and effects commonly require reconstruction and new encoding. Similarly, increasing resolution during transcoding does not recreate original detail. These distinctions matter when evaluating archives, comparing platform uploads, or diagnosing why a large output file still contains visible compression artifacts.

Licensing, deployment, and successor formats

AVC became deeply embedded in consumer and professional distribution. It is used in Blu-ray Disc video, digital television services, security cameras, video conferencing, and Internet streaming. Support in web browsers, smartphones, televisions, cameras, and operating-system media frameworks made it a practical common format across many otherwise different systems. Audio remains separate: an AVC video track can be paired with different audio coding formats according to the container and application.

The standard is associated with patented techniques. Patent pools and individual licensing arrangements have covered various forms of implementation and use, with terms varying by activity, jurisdiction, date, and contractual category. The availability of an open-source implementation does not by itself establish that all patent permissions are included. Conversely, patent licensing, software copyright licensing, and ownership of the audiovisual content are distinct matters within intellectual property. Compliance with one does not automatically settle the others.

Interoperability is also narrower than a format name suggests. A service may advertise H.264 support while accepting only 8-bit 4:2:0 streams, particular profile-level combinations, progressive pictures, or specific container conventions. Specialized 10-bit, 4:2:2, 4:4:4, scalable, or multiview streams may require different software or hardware. Color signaling, timestamps, rotation metadata, and parameter-set changes can further affect playback even when the compressed pictures are individually valid.

Later standards include High Efficiency Video Coding, or H.265/HEVC, first standardized in 2013, and Versatile Video Coding, or H.266/VVC, first standardized in 2020. The Alliance for Open Media released AV1 in 2018. These formats pursue improved compression and additional capabilities through more elaborate coding tools. Their advantages depend on content and implementation, while their deployment depends on hardware support, encoding cost, licensing arrangements, energy consumption, and compatible playback software.

Successor adoption does not require immediate elimination of AVC. Distribution systems may retain an AVC representation for older devices while offering newer formats to capable receivers. Real-time applications may continue using it because of predictable hardware behavior and broad interoperability. Long-lived recordings remain decodable independently of whether a newer standard offers better efficiency.

Operational maintenance also remains important for mature codecs. Compressed bitstreams can exercise complex parsing, buffer management, and reconstruction paths, so decoder implementations require continuing attention to security. Conformance testing, malformed-input testing, and protected execution environments address different parts of this problem. Dedicated hardware accelerators and software decoders may expose different capabilities and failure modes, making the exact playback implementation relevant even when every system identifies the content as H.264.