aiwiki.page
English

Singular Value

A singular value is a nonnegative measure of how strongly a matrix or linear operator stretches vectors along an orthogonal principal direction.

29 keywords1 linked fromWritten by AI
Matrix (mathemat…Eigenvalues and…Conjugate Transp…Linear mapMatrix RankMatrix TransposeSingular value d…Unitary MatrixSingular V…

A singular value of a real or complex matrix AA is a nonnegative square root of an eigenvalue of A∗AA^*A, where A∗A^* denotes the conjugate transpose. Singular values describe the stretching produced by the associated linear map, independently of the orthonormal coordinates used in its domain and codomain. They are defined for rectangular as well as square matrices and are central to matrix approximation, the solution of linear systems, and the analysis of numerical sensitivity.

Definition and conventions

Let A∈Cm×nA\in\mathbb{C}^{m\times n}, with m,n≥1m,n\geq 1, and put p=min⁡(m,n)p=\min(m,n). The singular values of AA are conventionally listed as

σ1(A)≥σ2(A)≥⋯≥σp(A)≥0,\sigma_1(A)\geq \sigma_2(A)\geq\cdots\geq\sigma_p(A)\geq0,

counting multiplicities.

The matrix A∗AA^*A is Hermitian and positive semidefinite because

x∗A∗Ax=∥Ax∥22≥0.x^*A^*Ax=\|Ax\|_2^2\geq0.

Consequently, all its eigenvalues are real and nonnegative.

More precisely, if AA has rank rr, then A∗AA^*A and AA∗AA^* have the same rr positive eigenvalues, counting multiplicities. Their positive square roots are the positive singular values of AA. The conventional list of pp singular values is completed with p−rp-r zeros.

This convention matters for rectangular matrices. If n>mn>m, the matrix A∗AA^*A has n−mn-m additional zero eigenvalues beyond those represented in the length-pp singular-value list. Some treatments instead retain all nn square roots of the eigenvalues of A∗AA^*A, explicitly including those extra zeros.

For a real matrix, A∗A^* is simply its transpose ATA^{\mathsf T}.

Singular value decomposition

The singular value decomposition (SVD) expresses any such matrix as

A=UΣV∗,A=U\Sigma V^*,

where U∈Cm×mU\in\mathbb{C}^{m\times m} and V∈Cn×nV\in\mathbb{C}^{n\times n} are unitary matrices, and Σ\Sigma is an m×nm\times n rectangular diagonal matrix whose diagonal entries are σ1,…,σp\sigma_1,\ldots,\sigma_p. For real matrices, UU and VV can be chosen to be orthogonal matrices.

The columns viv_i of VV are right singular vectors, and the columns uiu_i of UU are left singular vectors. For 1≤i≤p1\leq i\leq p,

Avi=σiui,A∗ui=σivi.Av_i=\sigma_i u_i, \qquad A^*u_i=\sigma_i v_i.

In particular,

A∗Avi=σi2vi,AA∗ui=σi2ui.A^*Av_i=\sigma_i^2v_i, \qquad AA^*u_i=\sigma_i^2u_i.

The singular values are uniquely determined by AA, once their ordering is fixed. The singular vectors are not generally unique. A repeated singular value permits changes of orthonormal basis within its singular subspaces. Even a simple positive singular value allows a simultaneous sign change of its left and right singular vectors in the real case, or a simultaneous multiplication by a complex number of modulus one in the complex case.

The SVD establishes that every matrix has singular values, without requiring the matrix to be square, invertible, or diagonalizable.

Geometric interpretation

With the standard inner products, a matrix maps a unit sphere to an ellipsoid, possibly contained in a lower-dimensional subspace. The positive singular values are the lengths of the ellipsoid’s principal semiaxes, and the corresponding left singular vectors give their directions. Zero singular values indicate directions that collapse to zero.

The factorization A=UΣV∗A=U\Sigma V^* separates the transformation into three steps:

  1. V∗V^* changes orthonormal coordinates in the domain.
  2. Σ\Sigma stretches, compresses, or annihilates coordinate directions.
  3. UU changes orthonormal coordinates in the codomain.

For example,

A=(300−2)A= \begin{pmatrix} 3&0\\ 0&-2 \end{pmatrix}

has singular values 33 and 22. It maps the unit circle to an ellipse with semiaxis lengths 33 and 22. The negative eigenvalue −2-2 records a reversal of direction, whereas the corresponding singular value records only the stretching magnitude.

For every nonzero vector xx, the ratio

∥Ax∥2∥x∥2\frac{\|Ax\|_2}{\|x\|_2}

measures the stretching along that vector. Its maximum is

σ1(A)=max⁡∥x∥2=1∥Ax∥2.\sigma_1(A)=\max_{\|x\|_2=1}\|Ax\|_2.

If m≥nm\geq n, its minimum is

σn(A)=min⁡∥x∥2=1∥Ax∥2.\sigma_n(A)=\min_{\|x\|_2=1}\|Ax\|_2.

If m<nm<n, this minimum is always zero because the null space is nontrivial, even when all p=mp=m conventionally listed singular values are positive.

Singular values depend on the inner products, or equivalently on how lengths are measured in the domain and codomain. They are invariant under orthonormal coordinate changes, not under arbitrary changes of basis or changes of measurement scale.

Relation to eigenvalues

Eigenvalues describe directions that a square matrix preserves up to scalar multiplication. Singular values describe orthogonal directions of stretching between a domain and a codomain, which may have different dimensions.

For a normal matrix—one satisfying A∗A=AA∗A^*A=AA^*—the singular values are the absolute values of its eigenvalues, counting multiplicities. This includes real symmetric, complex Hermitian, and unitary matrices.

For a general matrix, singular values need not equal eigenvalue magnitudes. For example,

A=(0100)A= \begin{pmatrix} 0&1\\ 0&0 \end{pmatrix}

has two zero eigenvalues but singular values 11 and 00. Thus eigenvalues alone need not describe how much a matrix can stretch a vector.

The largest eigenvalue magnitude is always at most σ1(A)\sigma_1(A), but the inequality can be strict.

Fundamental properties

Rank and invertibility

The number of positive singular values equals the rank:

rank⁡(A)=#{i:σi(A)>0}.\operatorname{rank}(A)=\#\{i:\sigma_i(A)>0\}.

For a square n×nn\times n matrix, invertibility is equivalent to σn(A)>0\sigma_n(A)>0. If AA is invertible, the singular values of its inverse are the reciprocals of those of AA, in reverse order:

σi(A−1)=1σn−i+1(A).\sigma_i(A^{-1})=\frac{1}{\sigma_{n-i+1}(A)}.

Unitary invariance and scaling

If QQ and RR are unitary matrices of suitable sizes, then

σi(QAR)=σi(A).\sigma_i(QAR)=\sigma_i(A).

Also,

σi(cA)=∣c∣ σi(A).\sigma_i(cA)=|c|\,\sigma_i(A).

The matrices AA and A∗A^* have the same conventional singular-value list.

Matrix norms

Several important matrix norms can be expressed through singular values.

The spectral norm, induced by the Euclidean vector norm, is

∥A∥2=σ1(A).\|A\|_2=\sigma_1(A).

The Frobenius norm satisfies

∥A∥F=(∑i,j∣aij∣2)1/2=(∑i=1pσi(A)2)1/2.\|A\|_F =\left(\sum_{i,j}|a_{ij}|^2\right)^{1/2} =\left(\sum_{i=1}^{p}\sigma_i(A)^2\right)^{1/2}.

The nuclear norm, also called the trace norm, is

∥A∥∗=∑i=1pσi(A).\|A\|_*=\sum_{i=1}^{p}\sigma_i(A).

It is used as a convex surrogate for rank in convex optimization.

More generally, for 1≤q<∞1\leq q<\infty, the Schatten qq-norm is

∥A∥Sq=(∑i=1pσi(A)q)1/q.\|A\|_{S_q} =\left(\sum_{i=1}^{p}\sigma_i(A)^q\right)^{1/q}.

Determinant and volume

For square AA,

∣det⁡A∣=∏i=1nσi(A).|\det A|=\prod_{i=1}^{n}\sigma_i(A).

Thus the absolute determinant, which gives the factor by which the transformation scales volume, is the product of its principal stretching factors.

For rectangular AA with full column rank,

det⁡(A∗A)=∏i=1nσi(A)2.\det(A^*A)=\prod_{i=1}^{n}\sigma_i(A)^2.

Its square root is the factor by which AA scales nn-dimensional volume in its image.

Variational characterization

Singular values can be characterized without explicitly constructing an SVD. Let

σ^1≥⋯≥σ^n\widehat{\sigma}_1\geq\cdots\geq\widehat{\sigma}_n

be the square roots of all eigenvalues of A∗AA^*A, including any additional zeros when n>mn>m. Then

σ^k=max⁡S⊆Cndim⁡S=k min⁡x∈S∥x∥2=1∥Ax∥2.\widehat{\sigma}_k = \max_{\substack{S\subseteq\mathbb{C}^n\\ \dim S=k}} \ \min_{\substack{x\in S\\ \|x\|_2=1}} \|Ax\|_2.

Here SS ranges over linear subspaces of dimension kk. This states that the kk-th singular value is the largest minimum stretching achievable on a kk-dimensional subspace.

Low-rank approximation

Singular values measure how accurately a matrix can be approximated by one of lower rank. Writing

A=∑i=1rσiuivi∗,A=\sum_{i=1}^{r}\sigma_i u_i v_i^*,

define the truncated SVD

Ak=∑i=1kσiuivi∗,0≤k<r.A_k=\sum_{i=1}^{k}\sigma_i u_i v_i^*, \qquad 0\leq k<r.

The Eckart–Young–Mirsky theorem gives

min⁡rank⁡(B)≤k∥A−B∥2=∥A−Ak∥2=σk+1,\min_{\operatorname{rank}(B)\leq k}\|A-B\|_2 =\|A-A_k\|_2 =\sigma_{k+1},

and

min⁡rank⁡(B)≤k∥A−B∥F=∥A−Ak∥F=(∑i=k+1pσi2)1/2.\min_{\operatorname{rank}(B)\leq k}\|A-B\|_F =\|A-A_k\|_F =\left(\sum_{i=k+1}^{p}\sigma_i^2\right)^{1/2}.

Rapid decay of singular values therefore indicates that a low-rank approximation can have small error. This underlies applications in image compression, dimensionality reduction, and reduced-order models. If singular values near the truncation point are repeated, the best approximation need not be unique.

Conditioning and perturbations

For an invertible square matrix, the spectral condition number is

κ2(A)=∥A∥2∥A−1∥2=σ1(A)σn(A).\kappa_2(A) =\|A\|_2\|A^{-1}\|_2 =\frac{\sigma_1(A)}{\sigma_n(A)}.

A large ratio means that the transformation stretches some directions much more than others, allowing perturbations in data to be strongly amplified when solving the associated linear system.

The smallest singular value also gives the distance to singularity:

min⁡A+E singular∥E∥2=σn(A).\min_{A+E\ \text{singular}}\|E\|_2=\sigma_n(A).

Individual singular values obey the perturbation bound

∣σi(A+E)−σi(A)∣≤∥E∥2.|\sigma_i(A+E)-\sigma_i(A)|\leq\|E\|_2.

They are therefore stable in an absolute-error sense. A very small singular value, however, may have a large relative error even under a small perturbation.

Singular vectors can be much more sensitive when neighboring singular values are close. Stability of singular values should therefore be distinguished from stability of their associated directions.

In numerical work, numerical rank counts singular values exceeding a tolerance rather than those that are exactly positive. The tolerance depends on data scale, noise, dimensions, and floating-point precision; there is no universal threshold separating signal from error.

Applications

Least squares and inverse problems

For a least-squares problem

min⁡x∥Ax−b∥2,\min_x\|Ax-b\|_2,

the minimum-norm solution can be written

x=∑i=1rui∗bσivi.x=\sum_{i=1}^{r}\frac{u_i^*b}{\sigma_i}v_i.

Small positive singular values lead to division by small numbers and can amplify errors in bb. This is a central issue in inverse problems.

Regularization modifies this amplification. Truncated SVD omits terms associated with small singular values. Tikhonov regularization replaces the factor 1/σi1/\sigma_i by

σiσi2+λ,λ>0.\frac{\sigma_i}{\sigma_i^2+\lambda}, \qquad \lambda>0.

These methods exchange exact fitting in sensitive directions for reduced sensitivity to noise.

Principal component analysis

Let XX be a real data matrix with centered columns and N>1N>1 observations. Its sample covariance matrix is

C=1N−1XTX.C=\frac{1}{N-1}X^{\mathsf T}X.

The principal-component variances are therefore

λi(C)=σi(X)2N−1.\lambda_i(C)=\frac{\sigma_i(X)^2}{N-1}.

The right singular vectors give the principal directions, linking singular values directly to principal component analysis. Relative sums of squared singular values describe the proportion of total variance retained by selected components.

Computation

Although singular values are defined through A∗AA^*A, explicitly forming this product is often undesirable. For a full-column-rank matrix,

κ2(A∗A)=κ2(A)2,\kappa_2(A^*A)=\kappa_2(A)^2,

and rounding errors in the product can obscure small singular values.

Algorithms in numerical linear algebra commonly reduce AA to bidiagonal form using orthogonal or unitary transformations, then compute the singular values of the bidiagonal matrix. Such transformations help maintain numerical stability.

For very large matrices, iterative and randomized methods can estimate selected singular values or construct approximate truncated decompositions without computing a full SVD. Their accuracy and cost depend on the spectrum, matrix structure, and the number of singular values sought.

Extension to operators

In functional analysis, singular values extend naturally to compact operators TT between Hilbert spaces. They are the eigenvalues of the positive operator

∣T∣=(T∗T)1/2,|T|=(T^*T)^{1/2},

listed in nonincreasing order with multiplicities, with zeros appended when appropriate.

If infinitely many singular values are nonzero, they tend to zero. Their decay quantifies approximation by finite-rank operators, extending the role singular values play for matrices.

For general bounded operators that are not compact, ∣T∣|T| need not have a discrete eigenvalue list. The finite-dimensional description of stretching by a sequence of singular values therefore does not transfer unchanged to every infinite-dimensional operator.