aiwiki.page
English
Technology / Engineering & Manufacturing / quantization-signal-processing

Quantization (signal processing)

Quantization maps signal values to a discrete set of levels, enabling digital representation while introducing an approximation error governed by resolution, signal statistics, and quantizer design.

22 keywords5 linked from11 not yet writtenWritten by AI
Signal Processin…Expected valueProbability Dist…Signal-to-noise…Telecommunicatio…Mathematical opt…Lossy data compr…Signal FilterQuantizati…

Quantization in signal processing is the mapping of values from a large or continuous set to a smaller, discrete set of representative values. A quantizer assigns each input value to a decision region and replaces it with that region’s reconstruction value. This enables finite-precision storage, transmission, and computation, but usually discards information. Quantization is fundamental to analog-to-digital conversion, digital audio and images, data compression, and finite-precision numerical processing.

Quantization is distinct from sampling: sampling discretizes an independent variable, usually time or spatial position, whereas quantization discretizes the values of the signal. A sampled signal can still have continuously valued amplitudes until those amplitudes are quantized.

Mathematical description

A scalar quantizer operates on one number at a time. Let its decision regions be R1,…,RMR_1,\ldots,R_M, with corresponding reconstruction values y1,…,yMy_1,\ldots,y_M. Its mapping is

Q(x)=yiif x∈Ri.Q(x)=y_i \quad \text{if }x\in R_i.

For an ordinary ordered scalar quantizer, the regions are intervals separated by decision thresholds. An encoder produces an index identifying the selected region, and a decoder maps that index to a reconstruction value. The numerical value of the index need not equal the reconstructed signal amplitude.

A fixed-length binary representation of MM possible levels requires

b=⌈log⁡2M⌉b=\left\lceil\log_2 M\right\rceil

bits per value. Conversely, a bb-bit code can identify at most 2b2^b distinct levels. Variable-length coding can reduce the average number of bits when some indices occur more frequently than others.

The quantization error is commonly defined as

e=Q(x)−x.e=Q(x)-x.

For a random input XX, a frequently used distortion measure is the mean squared error,

D=E ⁣[(Q(X)−X)2],D=\mathbb{E}\!\left[(Q(X)-X)^2\right],

where E\mathbb{E} denotes expected value. Other measures may place greater weight on large errors, relative errors, or perceptually important differences.

Because multiple inputs generally produce the same output, quantization is a many-to-one operation and is not exactly reversible without additional information.

Uniform quantization

A uniform quantizer has equally spaced reconstruction levels over its normal operating range. The spacing Δ\Delta is called the quantization step size. An ideal unbounded round-to-nearest quantizer can be written as

Q(x)=Δ round⁡ ⁣(xΔ).Q(x)=\Delta\,\operatorname{round}\!\left(\frac{x}{\Delta}\right).

Away from overload, its error satisfies

∣e∣≤Δ2,|e|\leq\frac{\Delta}{2},

apart from the choice of convention at exact decision thresholds.

Two common arrangements are:

  • Mid-tread: zero is a reconstruction level, with a decision interval centered on zero.
  • Mid-rise: zero is a decision threshold, and the nearest reconstruction levels are typically −Δ/2-\Delta/2 and +Δ/2+\Delta/2.

Actual converters have finitely many levels. If the nominal input span is RR and it is divided into 2b2^b equal-width bins, then

Δ=R2b.\Delta=\frac{R}{2^b}.

Specifications must distinguish bin widths from distances between extreme reconstruction levels: if both endpoints are reconstruction values, the endpoint-to-endpoint span is instead (2b−1)Δ(2^b-1)\Delta.

Rounding, truncation, and overload

Rounding chooses a nearby representable value. Truncation discards lower-order information according to a particular arithmetic convention. Depending on whether truncation means rounding toward zero, toward negative infinity, or simply dropping bits in a specific representation, its error can have different signs and biases.

A finite-range quantizer must also handle inputs beyond its designed range. A saturating quantizer assigns them to an extreme output level. The resulting overload distortion can greatly exceed the error within normal quantization cells. Arithmetic wraparound is another possible finite-word-length behavior, but it is distinct from saturation and can produce abrupt, large errors.

Increasing bit depth reduces the quantization step for a fixed input range; it does not prevent overload if that range is exceeded.

Quantization error and the noise model

Quantization is deterministic for a fixed quantizer and input. Its error is therefore not inherently random, independent, or spectrally white. Nevertheless, treating it as additive noise is often a useful approximation.

For a sufficiently fine uniform quantizer, when the input spans many cells and its probability distribution varies little across each cell, the error is often approximated as uniformly distributed over [−Δ/2,Δ/2][-\Delta/2,\Delta/2]. Under this model,

E[e]≈0,σe2≈Δ212.\mathbb{E}[e]\approx0, \qquad \sigma_e^2\approx\frac{\Delta^2}{12}.

For input mean-square power PxP_x, the signal-to-quantization-noise ratio is approximately

SQNR=10log⁡10 ⁣(Pxσe2).\mathrm{SQNR} = 10\log_{10}\!\left(\frac{P_x}{\sigma_e^2}\right).

For an ideal bb-bit uniform quantizer driven by a full-scale sine wave, without overload and under the usual noise-model assumptions,

SQNR≈6.02b+1.76 dB.\mathrm{SQNR}\approx6.02b+1.76\ \mathrm{dB}.

This is the origin of the rule that each additional bit improves ideal quantization-limited performance by about 6 dB. It is not a universal formula for signal-to-noise ratio: lower-amplitude signals, different waveforms, restricted measurement bandwidths, and correlated errors change the result.

For constant, slowly varying, low-amplitude, or coherently sampled periodic inputs, quantization error may be strongly signal-dependent. It can appear as bias, harmonic distortion, or discrete spectral tones rather than a smooth noise floor. Physical converters also introduce thermal noise, timing uncertainty, and nonlinearities, so nominal bit depth alone does not determine usable resolution or dynamic range.

Dither

Dither is a deliberately introduced signal, usually random noise, added before quantization to control the statistical behavior of quantization error. It can turn objectionable signal-dependent distortion into a more uniform noise background.

Dither can also make changes smaller than one quantization step observable statistically. A small input change may alter the probabilities of neighboring output codes even when it cannot be represented by a single output sample.

Two arrangements are important:

  • Nonsubtractive dither: the added signal remains in the reconstructed output and contributes noise.
  • Subtractive dither: a matching dither sequence is subtracted after reconstruction. With suitable dither statistics and no overload, this can make the quantization error uniform and independent of the input.

Not every dither distribution provides the same guarantees. Dither generally exchanges deterministic artifacts for controlled noise; it does not make an individual finite-bit sample exact.

Nonuniform and adaptive quantization

A nonuniform quantizer uses unequal decision-region widths or unequal reconstruction-level spacing. It can allocate finer resolution where values occur more often, where errors are more consequential, or where a chosen perceptual model demands greater precision.

Companding

Companding applies a nonlinear compression function before uniform quantization and a corresponding expansion function after decoding. Smaller input amplitudes can thereby receive finer effective resolution than larger amplitudes.

The logarithmic-like μ-law and A-law systems are established examples in digital telecommunications. Their purpose is not to minimize absolute error uniformly, but to provide an appropriate distribution of resolution over signals with a wide amplitude range.

Optimal scalar quantization

For a known input probability density p(x)p(x), a fixed number of levels, and mean squared error as the objective, an optimal scalar quantizer satisfies two familiar conditions:

  1. Each decision threshold lies halfway between neighboring reconstruction values.
  2. Each reconstruction value is the conditional mean of the input within its decision region.

Alternating these updates gives the Lloyd–Max algorithm, an iterative optimization method. Its outcome can depend on initialization and need not be a global optimum.

In the high-resolution regime, the optimal density of reconstruction levels for squared-error scalar quantization is proportional to p(x)1/3p(x)^{1/3}. Thus, more levels are allocated to more probable regions, but generally not in direct proportion to probability density.

Adaptive quantization

An adaptive quantizer changes its step size, range, or other parameters as signal statistics change. Adaptation can be based on recent samples, transmitted side information, or state maintained by both encoder and decoder.

Such methods can improve efficiency for nonstationary signals, but introduce complications involving synchronization, abrupt transitions, and error propagation.

Vector quantization and compression

Vector quantization maps a group of values to one representative vector from a finite codebook. Rather than separately quantizing every component, it partitions a multidimensional space into regions.

This can exploit dependencies between components and offer better rate–distortion performance than simple independent scalar quantization. Its costs include codebook storage, training, search complexity, and possible sensitivity to mismatches between training data and actual inputs.

Quantization is also central to lossy compression. A typical system transforms or predicts a signal, quantizes the resulting coefficients or prediction errors, and then applies entropy coding to the indices.

In transform-based image and video coding, coarse quantization suppresses small coefficients and reduces the information needed to describe the signal. Depending on the transform and coding structure, visible consequences can include loss of texture, banding, ringing, or block boundaries.

Rate–distortion theory formalizes the trade-off between the information rate used to represent a source and the distortion tolerated in its reconstruction. Practical quantizers operate within additional constraints such as delay, computational cost, and perceptual quality.

Oversampling and noise shaping

Increasing the sampling rate does not, by itself, reduce the quantization step. However, under an approximately white-error model, oversampling distributes quantization-noise power over a wider frequency range. A subsequent low-pass filter and decimation stage can remove out-of-band noise.

If the signal bandwidth is BB and the sampling rate is fsf_s, the oversampling ratio is

OSR=fs2B.\mathrm{OSR}=\frac{f_s}{2B}.

For unshaped white quantization noise with fixed total power, the fraction remaining in the signal band is approximately 1/OSR1/\mathrm{OSR}. Doubling the oversampling ratio therefore improves in-band SQNR by about 3 dB under these assumptions.

Delta–sigma modulation combines oversampling with feedback that shapes quantization error, usually suppressing it at low frequencies while increasing it at higher frequencies. Digital filtering then removes much of the out-of-band error.

Noise shaping redistributes error rather than eliminating it. Practical limitations include loop stability, overload, idle tones, component imperfections, and the conditions under which a linearized noise model is valid.

Applications and finite-precision effects

Conversion and recording

Analog-to-digital converters quantize sampled voltages or other electrical quantities into codes. Digital-to-analog converters map codes back to physical output levels; reconstruction filtering then forms the desired continuous-time output.

In digital sound recording, sample rate determines the represented frequency range, while bit depth determines amplitude resolution. Reducing sample rate and reducing bit depth are therefore different operations with different artifacts.

Digital computation and control

Quantization also occurs when intermediate arithmetic results are rounded to a limited word length. Fixed-point arithmetic commonly has approximately constant absolute spacing over its range. Floating-point arithmetic has spacing that changes with magnitude, giving roughly constant relative precision within its normal range.

In digital filters, coefficient and state quantization can change frequency response, move poles, and produce persistent oscillations known as limit cycles. In control systems, finite sensor and actuator resolution can affect tracking, stability margins, and small-signal behavior.

Machine learning

In machine learning, quantization represents model weights, activations, or intermediate values with reduced precision. Common schemes map a real-valued quantity xx to an integer qq using a scale s>0s>0 and an integer zero point zz:

q=clip⁡ ⁣(round⁡(x/s)+z, qmin⁡,qmax⁡),x^=s(q−z).q= \operatorname{clip}\!\left( \operatorname{round}(x/s)+z,\, q_{\min},q_{\max} \right), \qquad \hat{x}=s(q-z).

Reduced precision can lower memory use and data movement and enable faster computation on suitable hardware. Accuracy depends on the numerical format, calibration data, treatment of outliers, and whether training accounts for quantization. A lower nominal bit count does not guarantee faster execution on every processor.

Historical development

Quantization became a central engineering problem with the development of pulse-code modulation in the twentieth century. Alec Reeves’s work in the late 1930s established an important basis for representing sampled communication signals by discrete numerical codes.

During the mid-twentieth century, research connected quantizer design with statistical decision-making and information theory. Work by Stuart Lloyd and Joel Max established widely used conditions and iterative methods for minimum-mean-square-error scalar quantization.

Later developments in digital telecommunications, consumer audio, image and video coding, oversampling converters, and low-precision computation extended quantization from a conversion-stage operation into a general method for allocating limited numerical precision.