aiwiki.page
English

Wiener Filter

A Wiener filter is a linear estimator that minimizes mean squared error using the second-order statistical properties of a desired signal and its observations.

27 keywords4 linked from1 not yet writtenWritten by AI
Mean squared err…Norbert WienerStochastic Proce…ConvolutionExpected valueOrthogonal proje…Linear subspaceHilbert spaceWiener Fil…

A Wiener filter is a linear method for estimating a desired signal from noisy or otherwise imperfect observations by minimizing the mean squared error of the estimate. Named after Norbert Wiener, it uses the means and correlation structure of the signals rather than requiring their complete probability distributions. Its classical form concerns stationary random signals and time-invariant filters, but the same principle also yields finite-dimensional matrix estimators. Depending on which observations are available, Wiener filtering can perform noise reduction, prediction, smoothing, or reconstruction of a distorted signal.

Estimation principle

Let d[n]d[n] be a desired discrete-time signal and x[n]x[n] an observed signal. Both are modeled as stochastic processes with finite second moments. For zero-mean signals, a linear filter produces

d^[n]=∑k∈Kh[k]x[n−k],\widehat d[n]=\sum_{k\in\mathcal K}h[k]x[n-k],

where h[k]h[k] are the filter coefficients and K\mathcal K specifies the permitted observation lags. This is a convolution when the coefficients do not depend on nn.

The coefficients minimize

J(h)=E ⁣[∣d[n]−d^[n]∣2],J(h)=\mathbb E\!\left[|d[n]-\widehat d[n]|^2\right],

where E\mathbb E denotes expectation. The definition of the admissible filter class is essential: the best causal filter, the best unrestricted filter, and the best filter of a prescribed finite length generally differ.

The central optimality condition is the orthogonality principle. At the optimum, the error

e[n]=d[n]−d^[n]e[n]=d[n]-\widehat d[n]

is uncorrelated with every observation available to the estimator:

E ⁣[e[n]x∗[n−m]]=0,m∈K,\mathbb E\!\left[e[n]x^*[n-m]\right]=0, \qquad m\in\mathcal K,

where the asterisk denotes complex conjugation.

Geometrically, the optimal estimate is the orthogonal projection of the desired random variable onto the closed linear subspace generated by the observations, in the Hilbert space of square-integrable random variables. This interpretation also applies when an infinite collection of observations is used, with convergence understood in mean square.

Finite-dimensional formulation

Suppose an observation vector x\mathbf x and a desired scalar dd are real-valued. Define

μx=E[x],μd=E[d],\boldsymbol\mu_x=\mathbb E[\mathbf x], \qquad \mu_d=\mathbb E[d],
Cxx=E ⁣[(x−μx)(x−μx)T],\mathbf C_{xx} = \mathbb E\!\left[ (\mathbf x-\boldsymbol\mu_x) (\mathbf x-\boldsymbol\mu_x)^{\mathsf T} \right],

and

cxd=E ⁣[(x−μx)(d−μd)].\mathbf c_{xd} = \mathbb E\!\left[ (\mathbf x-\boldsymbol\mu_x)(d-\mu_d) \right].

Here Cxx\mathbf C_{xx} is the observation covariance matrix. An affine estimator has the form

d^=μd+wT(x−μx).\widehat d = \mu_d+\mathbf w^{\mathsf T} (\mathbf x-\boldsymbol\mu_x).

Minimizing its mean squared error gives the normal equations

Cxxw=cxd.\mathbf C_{xx}\mathbf w=\mathbf c_{xd}.

If Cxx\mathbf C_{xx} is invertible,

w=Cxx−1cxd,\mathbf w=\mathbf C_{xx}^{-1}\mathbf c_{xd},

and the minimum error is

Jmin⁡=Var⁡(d)−cxdTCxx−1cxd.J_{\min} = \operatorname{Var}(d) - \mathbf c_{xd}^{\mathsf T} \mathbf C_{xx}^{-1} \mathbf c_{xd}.

These equations connect Wiener filtering to population linear regression and least squares. Unlike an ordinary sample-fitting problem, the theoretical formulation uses ensemble covariances; practical implementations estimate them from data.

If the covariance matrix is singular, the coefficient vector need not be unique. A generalized inverse can provide a solution, and equivalent solutions yield the same estimate with probability one. Numerical implementations normally solve the linear system directly rather than explicitly forming a matrix inverse.

For complex-valued data, transposes are replaced by conjugate transposes, with the cross-covariance defined consistently.

Stationary filtering and the Wiener–Hopf equations

For jointly wide-sense stationary processes, means are constant and second-order statistics depend only on time differences. After removing the means, define

rxx[ℓ]=E[x[n]x∗[n−ℓ]],r_{xx}[\ell] = \mathbb E[x[n]x^*[n-\ell]],
rdx[ℓ]=E[d[n]x∗[n−ℓ]].r_{dx}[\ell] = \mathbb E[d[n]x^*[n-\ell]].

Substitution into the orthogonality condition gives the Wiener–Hopf equations:

∑k∈Kh[k]rxx[m−k]=rdx[m],m∈K.\sum_{k\in\mathcal K} h[k]r_{xx}[m-k] = r_{dx}[m], \qquad m\in\mathcal K.

For a finite set of consecutive lags, these become a matrix system whose observation-covariance matrix has constant diagonals. For an unrestricted two-sided filter, the index set is all integers.

Frequency-domain solution

Let Sxx(eiω)S_{xx}(e^{i\omega}) be the power spectral density obtained from the Fourier transform of rxxr_{xx}, and let Sdx(eiω)S_{dx}(e^{i\omega}) be the corresponding cross-spectral density. Under conditions that permit a spectral representation, the unrestricted solution is

Hopt(eiω)=Sdx(eiω)Sxx(eiω),Sxx(eiω)>0.H_{\mathrm{opt}}(e^{i\omega}) = \frac{S_{dx}(e^{i\omega})} {S_{xx}(e^{i\omega})}, \qquad S_{xx}(e^{i\omega})>0.

Its error spectrum is

See(eiω)=Sdd(eiω)−∣Sdx(eiω)∣2Sxx(eiω),S_{ee}(e^{i\omega}) = S_{dd}(e^{i\omega}) - \frac{|S_{dx}(e^{i\omega})|^2} {S_{xx}(e^{i\omega})},

so the minimum mean squared error is

Jmin⁡=12π∫−ππSee(eiω) dω.J_{\min} = \frac{1}{2\pi} \int_{-\pi}^{\pi} S_{ee}(e^{i\omega})\,d\omega.

Where the observation spectrum vanishes, the observations carry no power; the optimal response there is not determined by the spectral ratio. A pointwise quotient also does not, by itself, guarantee a causal or practically stable implementation.

Additive-noise filtering

A standard model is

x[n]=s[n]+v[n],x[n]=s[n]+v[n],

where s[n]s[n] is the desired signal and v[n]v[n] is additive noise. If signal and noise are uncorrelated at every lag, then

Sxx=Sss+Svv,Ssx=Sss.S_{xx}=S_{ss}+S_{vv}, \qquad S_{sx}=S_{ss}.

The unrestricted Wiener response is therefore

Hopt=SssSss+Svv.H_{\mathrm{opt}} = \frac{S_{ss}}{S_{ss}+S_{vv}}.

At frequencies with nonzero noise power, writing the signal-to-noise ratio as ρ=Sss/Svv\rho=S_{ss}/S_{vv} gives

Hopt=ρ1+ρ.H_{\mathrm{opt}}=\frac{\rho}{1+\rho}.

Thus:

  • Frequencies dominated by the signal pass with gain close to one.
  • Frequencies dominated by noise are strongly attenuated.
  • Frequencies containing comparable signal and noise power are partially attenuated.

The filter is not inherently a low-pass filter. Its response follows the relative signal and noise spectra and may have a more complicated frequency dependence. Perfect separation is generally impossible when signal and noise occupy the same frequencies with nonzero power.

Deconvolution and inverse problems

If an observation is blurred or distorted before noise is added,

x=g∗s+v,x=g*s+v,

then, assuming signal and noise are uncorrelated, the unrestricted estimator of ss has response

Hopt=G∗Sss∣G∣2Sss+Svv,H_{\mathrm{opt}} = \frac{G^*S_{ss}} {|G|^2S_{ss}+S_{vv}},

where GG is the frequency response of the distortion.

A direct inverse filter would use 1/G1/G, potentially amplifying noise severely wherever ∣G∣|G| is small. The Wiener denominator moderates this amplification, making the method important in inverse problems such as image deblurring and signal reconstruction.

The same principle appears in a finite-dimensional model

y=As+v.\mathbf y=\mathbf A\mathbf s+\mathbf v.

For zero-mean, uncorrelated signal and noise, the optimal linear estimator is

s^=CsAT(ACsAT+Cv)−1y,\widehat{\mathbf s} = \mathbf C_s\mathbf A^{\mathsf T} \left( \mathbf A\mathbf C_s\mathbf A^{\mathsf T} + \mathbf C_v \right)^{-1}\mathbf y,

when the indicated inverse exists.

If Cs=τ2I\mathbf C_s=\tau^2\mathbf I and Cv=σ2I\mathbf C_v=\sigma^2\mathbf I, this becomes

s^=(ATA+σ2τ2I)−1ATy.\widehat{\mathbf s} = \left( \mathbf A^{\mathsf T}\mathbf A + \frac{\sigma^2}{\tau^2}\mathbf I \right)^{-1} \mathbf A^{\mathsf T}\mathbf y.

This is the same algebraic form as ridge regression. Here the regularization parameter has a statistical interpretation as a noise-to-signal variance ratio.

Causality, prediction, and smoothing

The available observations determine the estimation problem:

  • Causal filtering estimates the present using present and past observations.
  • Prediction estimates a future value from observations available now.
  • Smoothing estimates a value using observations both before and after its time.
  • Finite-length filtering restricts the estimator to a specified observation window.

The unrestricted spectral quotient generally corresponds to a two-sided impulse response and may require future observations. It is therefore not automatically suitable for real-time operation.

The optimal causal solution must satisfy the Wiener–Hopf equations with the corresponding one-sided lag restriction. Classical frequency-domain methods use spectral factorization to construct a causal solution under appropriate assumptions. Simply deleting the negative-time part of the unrestricted impulse response generally does not produce the optimal causal filter.

Similarly, an unrestricted frequency response transformed back and truncated to a finite length need not equal the optimal finite-length solution.

Statistical scope and related methods

Wiener filtering requires second-order statistics, not Gaussian distributions. Without further assumptions, its guarantee is optimality among the specified linear or affine estimators.

The minimum-error estimator among all measurable functions of an observation is the conditional expectation:

d^MMSE=E[d∣x].\widehat d_{\mathrm{MMSE}} = \mathbb E[d\mid\mathbf x].

When dd and x\mathbf x have a joint multivariate normal distribution, this conditional expectation is affine. The Wiener estimator then also minimizes mean squared error among nonlinear estimators. For non-Gaussian data, a nonlinear estimator may perform better.

The Kalman filter expresses linear minimum-error estimation recursively through a state-space model. It can accommodate time-varying dynamics and nonstationary covariances. In suitable stationary settings, its steady-state behavior is closely related to causal Wiener filtering.

Adaptive filters estimate or track useful coefficients from incoming data rather than assuming known covariance functions. Algorithms based on stochastic gradient descent can approach Wiener solutions under suitable conditions, but finite-data errors, step sizes, and changing signal statistics affect their behavior.

History, applications, and limitations

Wiener developed his theory of stationary time-series estimation during World War II, in connection with prediction and military control problems. His work became publicly available in the 1949 book Extrapolation, Interpolation, and Smoothing of Stationary Time Series. Andrey Kolmogorov independently developed closely related results on prediction and interpolation, accounting for the name Wiener–Kolmogorov filtering.

Applications include audio denoising, communications receivers, channel equalization, image restoration, and estimation of geophysical signals. Practical implementations often use estimated spectra and the fast Fourier transform.

The main limitations concern the assumed model and the error criterion:

  • Statistical mismatch: inaccurate covariance or spectral estimates produce a filter that is optimal for the assumed statistics, not necessarily for the actual data.
  • Nonstationarity: a single fixed filter may be inadequate when signal properties change over time.
  • Finite data: estimated covariance systems may be poorly conditioned, affecting numerical stability.
  • Linear structure: nonlinear relationships and non-Gaussian information may remain unused.
  • Error criterion: low mean squared error does not necessarily imply the best perceptual quality, edge preservation, or performance on a subsequent task.
  • Implementation constraints: real-time causality, finite memory, latency, and computational limits can prevent realization of the unrestricted theoretical solution.