Numerical differentiation is the computation of approximate derivatives from values of a function at discrete points. It is used when an explicit derivative is unavailable, inconvenient to evaluate, or less useful than an estimate derived directly from measurements. A central topic in numerical analysis, it includes finite-difference formulas, differentiation of interpolants and fitted curves, spectral methods, and techniques designed for noisy data. Its defining difficulty is that differentiation amplifies small variations: reducing the spacing between samples can improve approximation accuracy while simultaneously increasing the influence of roundoff error or measurement noise.
Basic finite-difference formulas
For a sufficiently smooth function , its derivative at is defined by a limit of difference quotients. Numerical differentiation replaces that limit with a computation using a nonzero step size .
The simplest formulas are the forward difference,
and the backward difference,
Expanding the sampled values in a Taylor series gives
Both therefore have first-order truncation error: their leading approximation error is proportional to , provided the relevant derivatives remain bounded. Here, big-O notation describes the asymptotic size of the error as approaches zero.
The central difference uses samples on both sides:
Its symmetric arrangement cancels the leading first-order error term:
It is consequently second-order accurate. Central formulas are often more accurate than first-order one-sided formulas for smooth functions, but cannot be applied directly at a boundary where values on one side are unavailable.
A second-order forward formula is
Higher derivatives can also be estimated. The standard central formula for the second derivative is
with truncation error . These formulas also supply the local derivative approximations used in the finite difference method for differential equations.
Stencils and construction of formulas
A stencil is the collection of sample locations and weights used in a derivative approximation. A general formula for the -th derivative has the form
where the specify sample offsets and the are dimensionless weights.
Taylor expansion shows that a formula has error , under suitable smoothness assumptions, if its moments satisfy
These conditions make the formula exact for the corresponding low-degree polynomials. Increasing the number of sample points can permit a higher-order formula. For example, a five-point central stencil gives
whose truncation error is .
Another construction uses interpolation: fit a polynomial through the sampled values and differentiate that polynomial at the desired point. This approach naturally accommodates unequal sample spacing. For distinct nodes , write the interpolant as
where are the Lagrange basis polynomials. Then
High formal order does not by itself guarantee better results. Wider stencils can span regions where local smoothness assumptions are poor, and their weights can amplify errors in the sampled values.
Truncation error, roundoff, and step size
Two principal errors compete when differentiating a computer-evaluated function:
- Truncation error, caused by replacing a limiting operation with a finite formula.
- Evaluation and roundoff error, arising from imperfect function values and floating-point arithmetic.
If each function evaluation has an absolute error of characteristic size , a first-derivative difference formula typically amplifies that error by a factor proportional to . A useful approximate error model is
where is the truncation order and depend on the function and stencil.
The first term decreases as the step becomes smaller; the second increases. When and are positive, minimizing this model gives
Under common scaling assumptions, if is proportional to machine precision , the characteristic step scales as
These are scaling laws, not universal step-size prescriptions. The appropriate dimensional step also depends on the scales of the input, function values, and higher derivatives.
When is very small, subtracting nearly equal function values causes cancellation, exposing errors already present in those values. Eventually, the floating-point representation of may even equal that of . Thus, taking a smaller step does not necessarily produce a more accurate derivative.
For an -th derivative, perturbations in the sampled values are generally amplified on the scale of . Higher derivatives are therefore especially demanding. This sensitivity is central to the numerical stability of differentiation algorithms.
Extrapolation and adaptive estimation
Richardson extrapolation improves an approximation by canceling a known leading error term. If
then
cancels the term . For a central first difference, whose smooth-function expansion contains even powers of , this construction raises the truncation order from two to four.
Repeated extrapolation can produce high-order estimates from a sequence of step sizes. Adaptive algorithms compare such estimates, assess their consistency, and stop when an error criterion is met or further refinement ceases to help.
These procedures depend on the assumed error expansion being valid. Roundoff, noise, discontinuities, and nonsmooth behavior can prevent estimates from entering the expected asymptotic regime. Agreement between two estimates is evidence of consistency, not a general proof of accuracy.
Differentiation of noisy data
For measured samples
the perturbations need not decrease when sample spacing decreases. Numerical differentiation can therefore become less reliable on a finer grid unless the noise is also controlled.
For example, if the errors at and are independent, each with mean zero and variance , the noise contribution to the central first-difference estimate has variance
This growth reflects a deeper issue: differentiation is an unbounded operation under common norms on function values. A perturbation
has amplitude , but its derivative has amplitude . Arbitrarily small, rapidly oscillating perturbations can consequently produce large derivative errors. Recovering derivatives from noisy observations is therefore often treated as an inverse problem.
Common approaches include:
- Local polynomial fitting: fit a low-degree polynomial over a neighborhood by least squares, then differentiate the fitted polynomial. The neighborhood supplies averaging rather than exact interpolation through every noisy point.
- Smoothing splines: fit a smooth curve while penalizing excessive roughness, then differentiate the fitted curve.
- Regularized derivative recovery: estimate a derivative by balancing agreement with observations against a smoothness or structural penalty.
- Frequency-domain filtering: suppress poorly resolved high-frequency components before or during differentiation.
These approaches introduce regularization and a bias–variance tradeoff. Stronger smoothing reduces noise amplification but can attenuate genuine rapid changes. The smoothing scale is distinct from the sample spacing, although the two interact.
Spectral differentiation
For smooth periodic functions, derivatives can be computed from Fourier coefficients. If
then, when termwise differentiation is justified,
For sampled data, a discrete Fourier transform supplies coefficients for a finite trigonometric representation. Differentiation multiplies each coefficient by its frequency factor; an inverse transform returns derivative values at the sample points. The fast Fourier transform makes this efficient.
Spectral differentiation can converge very rapidly for smooth periodic functions, with exponential convergence under suitable analyticity assumptions. Its limitations include high-frequency noise amplification, sampling and aliasing errors, and boundary artifacts when a nonperiodic function is represented as periodic. Nonperiodic spectral methods instead use other bases, including suitable orthogonal polynomials.
Complex-step differentiation
Complex-step differentiation estimates the derivative of a real-valued function that admits an appropriate analytic extension and can be evaluated with complex arguments. For real ,
so
with truncation error .
Unlike ordinary differences, this formula does not subtract nearby real function values. In a compatible implementation, it therefore avoids the principal cancellation mechanism of finite differences and permits very small steps.
Its applicability is restricted. Operations such as absolute values, complex conjugation, extraction of real parts, or some branch-dependent computations can invalidate the required analytic behavior. Numerical code must also preserve the small imaginary component throughout the evaluation.
Multivariable derivatives and applications
For a scalar function , finite differences estimate components of the gradient by perturbing individual coordinates. For a vector-valued function , the same procedure estimates columns of its Jacobian matrix:
where is the -th coordinate vector. Directional perturbations can estimate products without forming the full matrix.
Forward differences require one additional evaluation per input coordinate after the base value is available; central differences generally require two. Known sparsity can sometimes reduce this cost by allowing carefully chosen simultaneous perturbations.
Applications include mathematical optimization, sensitivity analysis, derivative checks for software, estimation of rates from experimental observations, and discretization of partial differential equations.
Numerical differentiation differs from automatic differentiation. Automatic differentiation propagates derivatives through a program using the chain rule, rather than estimating them from finitely separated input values. It avoids finite-difference truncation error but requires access to a suitable computational representation. Numerical differentiation remains useful for black-box functions and observed data.
Interpretation and limitations
A numerical derivative is an estimate at a particular resolution, not a substitute for establishing differentiability. At a corner, discontinuity, or unresolved transition, a finite formula may return a value even though the corresponding classical derivative does not exist.
For example, the central difference of at zero is zero for every positive , despite the function having no derivative there. One-sided differences reveal the incompatible slopes.
Reliability therefore depends on more than a formula’s formal order. Relevant factors include local smoothness, sample geometry, boundary access, function-evaluation error, noise level, and the physical or computational scale at which a derivative is meaningful.