Singular value decomposition (SVD) is a matrix factorization that expresses a rectangular matrix as the product of two orthogonal or unitary matrices and a diagonal matrix of nonnegative numbers. These numbers, called singular values, describe how strongly the matrix stretches independent directions. A fundamental construction in linear algebra and numerical linear algebra, SVD exists for every finite real or complex matrix, including matrices that are rectangular or rank-deficient. (netlib.org)
Definition and forms
For (A\in\mathbb C^{m\times n}), a full SVD has the form
[ A=U\Sigma V^*, ]
where (U\in\mathbb C^{m\times m}) and (V\in\mathbb C^{n\times n}) are unitary matrices, and (V^*) denotes the conjugate transpose. Thus (U^*U=I_m) and (V^*V=I_n), with (I_m) and (I_n) denoting identity matrices. The rectangular diagonal matrix (\Sigma) contains the singular values
[ \sigma_1\geq\sigma_2\geq\cdots\geq\sigma_p\geq0, \qquad p=\min(m,n). ]
For real matrices, (U) and (V) are orthogonal matrices, and the conjugate transpose becomes the ordinary transpose. The columns (u_i) and (v_i) are left and right singular vectors, satisfying (Av_i=\sigma_i u_i) and (A^*u_i=\sigma_i v_i) for (1\leq i\leq p). (netlib.org)
An economy-size SVD retains only the first (p) columns of each factor and a (p\times p) diagonal matrix. A compact, rank-(r) SVD retains only the (r) positive singular values and their vectors. Both remain exact factorizations; a truncated SVD that discards positive singular values is instead an approximation. Terminology such as “thin” and “reduced” varies between texts and software. (mpi-magdeburg.mpg.de)
Geometry and relation to eigenvalues
SVD describes a linear map between two vector spaces using orthonormal bases. For real matrices, (V^T) changes the input coordinates, (\Sigma) stretches or collapses coordinate directions, and (U) changes the output coordinates. Orthogonal factors preserve lengths and angles; geometrically, the image of the unit ball is an ellipsoid, possibly flattened into a lower-dimensional subspace. Its nonzero semiaxis lengths are the positive singular values. (heath.cs.illinois.edu)
The decomposition is closely related to eigenvalues and eigenvectors:
[ A^A=V(\Sigma^\Sigma)V^, \qquad AA^=U(\Sigma\Sigma^)U^. ]
Consequently, the positive singular values are square roots of the nonzero eigenvalues of either matrix. Their existence follows from the spectral theorem for Hermitian matrices. Unlike matrix diagonalization, SVD uses separate input and output bases and requires neither a square matrix nor a basis of eigenvectors of (A) itself. (nlp.stanford.edu)
The ordered singular values are unique, but the vectors need not be. Corresponding vector pairs can change sign, or share a complex phase, without changing (A). Repeated singular values permit simultaneous changes of orthonormal basis within the corresponding singular subspaces. (arxiv.org)
Rank, norms, and conditioning
The rank of (A) equals the number of positive singular values. In a full decomposition, the first (r) left singular vectors span the column space; the remaining right singular vectors span the null space of (A). These relationships make SVD a systematic description of the matrix’s fundamental subspaces. (prime.pages.ewi.tudelft.nl)
Singular values also determine important matrix norms:
[ |A|_2=\sigma_1, \qquad |A|F=\left(\sum{i=1}^{p}\sigma_i^2\right)^{1/2}. ]
For a nonsingular square matrix, its spectral condition number is (\kappa_2(A)=\sigma_1/\sigma_n). A large ratio indicates sensitivity in solving linear systems. In floating-point computation, numerical rank is assessed using a tolerance rather than exact equality to zero, because rounding and measurement errors can obscure small singular values. (mpi-magdeburg.mpg.de)
Low-rank approximation
For (k<r), the truncated decomposition is
[ A_k=\sum_{i=1}^{k}\sigma_i u_i v_i^*. ]
The Eckart–Young–Mirsky theorem states that this is a best low-rank approximation among matrices of rank at most (k), in both spectral and Frobenius norms. Its errors are
[ |A-A_k|2=\sigma{k+1}, \qquad |A-A_k|F= \left(\sum{i=k+1}^{p}\sigma_i^2\right)^{1/2}. ]
Rapidly decreasing singular values therefore indicate that comparatively few terms can approximate the matrix accurately. The retained factors require (k(m+n+1)) scalar entries rather than (mn), which can support data compression when (k) is sufficiently small. (mpi-magdeburg.mpg.de)
Least squares and data analysis
The Moore–Penrose pseudoinverse follows directly:
[ A^+=V\Sigma^+U^*, ]
where (\Sigma^+) reciprocates positive singular values, leaves zeros unchanged, and reverses the rectangular dimensions. For the least-squares problem (\min_x|Ax-b|_2), (x=A^+b) is the solution of minimum Euclidean norm. This remains meaningful for inconsistent or underdetermined systems of linear equations. Small singular values can amplify errors when reciprocated, motivating truncation and regularization in inverse computations. (mpi-magdeburg.mpg.de)
In principal component analysis, let (X) contain (N>1) observations as rows and mean-centered variables as columns. Its right singular vectors are principal directions, and the eigenvalues of the sample covariance matrix (X^TX/(N-1)) are (\sigma_i^2/(N-1)). Principal-component scores are (XV=U\Sigma). This connects SVD to dimensionality reduction, while distinguishing centered PCA from decomposition of uncentered data. (mit.edu)
Numerical computation
Practical dense-matrix algorithms usually reduce (A) to bidiagonal form through orthogonal transformations, then compute the bidiagonal SVD using QR iteration or divide-and-conquer methods. Explicitly forming (A^*A), although useful theoretically, can lose accuracy for small singular values. For large-scale approximation, randomized methods first identify a smaller subspace capturing dominant directions and then decompose the projected matrix. (netlib.org)