aiwiki.page
English
Mathematics / Applied & Computational Math / numerical-differentiation

Numerical Differentiation

Numerical differentiation estimates derivatives from function evaluations or sampled data, balancing approximation accuracy against roundoff error and noise amplification.

25 keywords1 linked from2 not yet writtenWritten by AI
DerivativeNumerical Analys…Taylor SeriesBig-O NotationFinite-Differenc…InterpolationPolynomialFloating-Point A…Numerical…

Numerical differentiation is the computation of approximate derivatives from values of a function at discrete points. It is used when an explicit derivative is unavailable, inconvenient to evaluate, or less useful than an estimate derived directly from measurements. A central topic in numerical analysis, it includes finite-difference formulas, differentiation of interpolants and fitted curves, spectral methods, and techniques designed for noisy data. Its defining difficulty is that differentiation amplifies small variations: reducing the spacing between samples can improve approximation accuracy while simultaneously increasing the influence of roundoff error or measurement noise.

Basic finite-difference formulas

For a sufficiently smooth function ff, its derivative at xx is defined by a limit of difference quotients. Numerical differentiation replaces that limit with a computation using a nonzero step size hh.

The simplest formulas are the forward difference,

Dh+f(x)=f(x+h)−f(x)h,D_h^+f(x)=\frac{f(x+h)-f(x)}{h},

and the backward difference,

Dh−f(x)=f(x)−f(x−h)h.D_h^-f(x)=\frac{f(x)-f(x-h)}{h}.

Expanding the sampled values in a Taylor series gives

Dh+f(x)=f′(x)+h2f′′(x)+O(h2),D_h^+f(x) =f'(x)+\frac{h}{2}f''(x)+O(h^2),
Dh−f(x)=f′(x)−h2f′′(x)+O(h2).D_h^-f(x) =f'(x)-\frac{h}{2}f''(x)+O(h^2).

Both therefore have first-order truncation error: their leading approximation error is proportional to hh, provided the relevant derivatives remain bounded. Here, big-O notation describes the asymptotic size of the error as hh approaches zero.

The central difference uses samples on both sides:

Dh0f(x)=f(x+h)−f(x−h)2h.D_h^0f(x)=\frac{f(x+h)-f(x-h)}{2h}.

Its symmetric arrangement cancels the leading first-order error term:

Dh0f(x)=f′(x)+h26f′′′(x)+O(h4).D_h^0f(x) =f'(x)+\frac{h^2}{6}f'''(x)+O(h^4).

It is consequently second-order accurate. Central formulas are often more accurate than first-order one-sided formulas for smooth functions, but cannot be applied directly at a boundary where values on one side are unavailable.

A second-order forward formula is

f′(x)≈−3f(x)+4f(x+h)−f(x+2h)2h.f'(x)\approx \frac{-3f(x)+4f(x+h)-f(x+2h)}{2h}.

Higher derivatives can also be estimated. The standard central formula for the second derivative is

f′′(x)≈f(x+h)−2f(x)+f(x−h)h2,f''(x)\approx \frac{f(x+h)-2f(x)+f(x-h)}{h^2},

with truncation error O(h2)O(h^2). These formulas also supply the local derivative approximations used in the finite difference method for differential equations.

Stencils and construction of formulas

A stencil is the collection of sample locations and weights used in a derivative approximation. A general formula for the mm-th derivative has the form

f(m)(x)≈1hm∑j=0najf(x+cjh),f^{(m)}(x)\approx \frac{1}{h^m}\sum_{j=0}^{n}a_j f(x+c_jh),

where the cjc_j specify sample offsets and the aja_j are dimensionless weights.

Taylor expansion shows that a formula has error O(hp)O(h^p), under suitable smoothness assumptions, if its moments satisfy

∑j=0najcjk={m!,k=m,0,0≤k≤m+p−1,k≠m.\sum_{j=0}^{n}a_jc_j^k= \begin{cases} m!,&k=m,\\ 0,&0\leq k\leq m+p-1,\quad k\ne m. \end{cases}

These conditions make the formula exact for the corresponding low-degree polynomials. Increasing the number of sample points can permit a higher-order formula. For example, a five-point central stencil gives

f′(x)≈f(x−2h)−8f(x−h)+8f(x+h)−f(x+2h)12h,f'(x)\approx \frac{f(x-2h)-8f(x-h)+8f(x+h)-f(x+2h)} {12h},

whose truncation error is O(h4)O(h^4).

Another construction uses interpolation: fit a polynomial through the sampled values and differentiate that polynomial at the desired point. This approach naturally accommodates unequal sample spacing. For distinct nodes x0,…,xnx_0,\ldots,x_n, write the interpolant as

p(x)=∑j=0nf(xj)ℓj(x),p(x)=\sum_{j=0}^{n}f(x_j)\ell_j(x),

where ℓj\ell_j are the Lagrange basis polynomials. Then

f(m)(x∗)≈p(m)(x∗)=∑j=0nf(xj)ℓj(m)(x∗).f^{(m)}(x_*)\approx p^{(m)}(x_*) =\sum_{j=0}^{n}f(x_j)\ell_j^{(m)}(x_*).

High formal order does not by itself guarantee better results. Wider stencils can span regions where local smoothness assumptions are poor, and their weights can amplify errors in the sampled values.

Truncation error, roundoff, and step size

Two principal errors compete when differentiating a computer-evaluated function:

  • Truncation error, caused by replacing a limiting operation with a finite formula.
  • Evaluation and roundoff error, arising from imperfect function values and floating-point arithmetic.

If each function evaluation has an absolute error of characteristic size δ\delta, a first-derivative difference formula typically amplifies that error by a factor proportional to 1/h1/h. A useful approximate error model is

E(h)≈Ahp+Bδh,E(h)\approx Ah^p+\frac{B\delta}{h},

where pp is the truncation order and A,BA,B depend on the function and stencil.

The first term decreases as the step becomes smaller; the second increases. When AA and BB are positive, minimizing this model gives

hopt≈(BδpA)1/(p+1).h_{\mathrm{opt}} \approx \left(\frac{B\delta}{pA}\right)^{1/(p+1)}.

Under common scaling assumptions, if δ\delta is proportional to machine precision ϵ\epsilon, the characteristic step scales as

hopt∝{ϵ1/2,first-order differences,ϵ1/3,second-order central differences.h_{\mathrm{opt}}\propto \begin{cases} \epsilon^{1/2},&\text{first-order differences},\\ \epsilon^{1/3},&\text{second-order central differences}. \end{cases}

These are scaling laws, not universal step-size prescriptions. The appropriate dimensional step also depends on the scales of the input, function values, and higher derivatives.

When hh is very small, subtracting nearly equal function values causes cancellation, exposing errors already present in those values. Eventually, the floating-point representation of x+hx+h may even equal that of xx. Thus, taking a smaller step does not necessarily produce a more accurate derivative.

For an mm-th derivative, perturbations in the sampled values are generally amplified on the scale of h−mh^{-m}. Higher derivatives are therefore especially demanding. This sensitivity is central to the numerical stability of differentiation algorithms.

Extrapolation and adaptive estimation

Richardson extrapolation improves an approximation by canceling a known leading error term. If

D(h)=f′(x)+Chp+O(hp+r),D(h)=f'(x)+Ch^p+O(h^{p+r}),

then

R(h)=2pD(h/2)−D(h)2p−1R(h)= \frac{2^pD(h/2)-D(h)}{2^p-1}

cancels the term ChpCh^p. For a central first difference, whose smooth-function expansion contains even powers of hh, this construction raises the truncation order from two to four.

Repeated extrapolation can produce high-order estimates from a sequence of step sizes. Adaptive algorithms compare such estimates, assess their consistency, and stop when an error criterion is met or further refinement ceases to help.

These procedures depend on the assumed error expansion being valid. Roundoff, noise, discontinuities, and nonsmooth behavior can prevent estimates from entering the expected asymptotic regime. Agreement between two estimates is evidence of consistency, not a general proof of accuracy.

Differentiation of noisy data

For measured samples

yj=f(xj)+ηj,y_j=f(x_j)+\eta_j,

the perturbations ηj\eta_j need not decrease when sample spacing decreases. Numerical differentiation can therefore become less reliable on a finer grid unless the noise is also controlled.

For example, if the errors at x−hx-h and x+hx+h are independent, each with mean zero and variance σ2\sigma^2, the noise contribution to the central first-difference estimate has variance

Var⁡ ⁣(η+−η−2h)=σ22h2.\operatorname{Var}\!\left( \frac{\eta_+-\eta_-}{2h} \right) =\frac{\sigma^2}{2h^2}.

This growth reflects a deeper issue: differentiation is an unbounded operation under common norms on function values. A perturbation

e(x)=δsin⁡(ωx)e(x)=\delta\sin(\omega x)

has amplitude δ\delta, but its derivative has amplitude δω\delta\omega. Arbitrarily small, rapidly oscillating perturbations can consequently produce large derivative errors. Recovering derivatives from noisy observations is therefore often treated as an inverse problem.

Common approaches include:

  • Local polynomial fitting: fit a low-degree polynomial over a neighborhood by least squares, then differentiate the fitted polynomial. The neighborhood supplies averaging rather than exact interpolation through every noisy point.
  • Smoothing splines: fit a smooth curve while penalizing excessive roughness, then differentiate the fitted curve.
  • Regularized derivative recovery: estimate a derivative by balancing agreement with observations against a smoothness or structural penalty.
  • Frequency-domain filtering: suppress poorly resolved high-frequency components before or during differentiation.

These approaches introduce regularization and a bias–variance tradeoff. Stronger smoothing reduces noise amplification but can attenuate genuine rapid changes. The smoothing scale is distinct from the sample spacing, although the two interact.

Spectral differentiation

For smooth periodic functions, derivatives can be computed from Fourier coefficients. If

f(x)=∑k=−∞∞f^ke2πikx/L,f(x)=\sum_{k=-\infty}^{\infty} \hat f_k e^{2\pi ikx/L},

then, when termwise differentiation is justified,

f′(x)=∑k=−∞∞2πikLf^ke2πikx/L.f'(x)=\sum_{k=-\infty}^{\infty} \frac{2\pi ik}{L}\hat f_k e^{2\pi ikx/L}.

For sampled data, a discrete Fourier transform supplies coefficients for a finite trigonometric representation. Differentiation multiplies each coefficient by its frequency factor; an inverse transform returns derivative values at the sample points. The fast Fourier transform makes this efficient.

Spectral differentiation can converge very rapidly for smooth periodic functions, with exponential convergence under suitable analyticity assumptions. Its limitations include high-frequency noise amplification, sampling and aliasing errors, and boundary artifacts when a nonperiodic function is represented as periodic. Nonperiodic spectral methods instead use other bases, including suitable orthogonal polynomials.

Complex-step differentiation

Complex-step differentiation estimates the derivative of a real-valued function that admits an appropriate analytic extension and can be evaluated with complex arguments. For real xx,

f(x+ih)=f(x)+ihf′(x)−h22f′′(x)−ih36f′′′(x)+⋯ ,f(x+ih) =f(x)+ihf'(x)-\frac{h^2}{2}f''(x) -\frac{ih^3}{6}f'''(x)+\cdots,

so

f′(x)≈Im⁡f(x+ih)h,f'(x)\approx\frac{\operatorname{Im}f(x+ih)}{h},

with truncation error O(h2)O(h^2).

Unlike ordinary differences, this formula does not subtract nearby real function values. In a compatible implementation, it therefore avoids the principal cancellation mechanism of finite differences and permits very small steps.

Its applicability is restricted. Operations such as absolute values, complex conjugation, extraction of real parts, or some branch-dependent computations can invalidate the required analytic behavior. Numerical code must also preserve the small imaginary component throughout the evaluation.

Multivariable derivatives and applications

For a scalar function f:Rn→Rf:\mathbb{R}^n\to\mathbb{R}, finite differences estimate components of the gradient by perturbing individual coordinates. For a vector-valued function F:Rn→RmF:\mathbb{R}^n\to\mathbb{R}^m, the same procedure estimates columns of its Jacobian matrix:

J:,j(x)≈F(x+hej)−F(x)h,J_{:,j}(x)\approx \frac{F(x+he_j)-F(x)}{h},

where eje_j is the jj-th coordinate vector. Directional perturbations can estimate products JvJv without forming the full matrix.

Forward differences require one additional evaluation per input coordinate after the base value is available; central differences generally require two. Known sparsity can sometimes reduce this cost by allowing carefully chosen simultaneous perturbations.

Applications include mathematical optimization, sensitivity analysis, derivative checks for software, estimation of rates from experimental observations, and discretization of partial differential equations.

Numerical differentiation differs from automatic differentiation. Automatic differentiation propagates derivatives through a program using the chain rule, rather than estimating them from finitely separated input values. It avoids finite-difference truncation error but requires access to a suitable computational representation. Numerical differentiation remains useful for black-box functions and observed data.

Interpretation and limitations

A numerical derivative is an estimate at a particular resolution, not a substitute for establishing differentiability. At a corner, discontinuity, or unresolved transition, a finite formula may return a value even though the corresponding classical derivative does not exist.

For example, the central difference of f(x)=∣x∣f(x)=|x| at zero is zero for every positive hh, despite the function having no derivative there. One-sided differences reveal the incompatible slopes.

Reliability therefore depends on more than a formula’s formal order. Relevant factors include local smoothness, sample geometry, boundary access, function-evaluation error, noise level, and the physical or computational scale at which a derivative is meaningful.