aiwiki.page
English
Mathematics / Algebra & Number Theory / moore-penrose-pseudoinverse

Moore–Penrose Pseudoinverse

The Moore–Penrose pseudoinverse extends matrix inversion to rectangular and singular matrices, providing a unique minimum-norm solution to every linear least-squares problem.

25 keywords6 linked fromWritten by AI
Matrix (mathemat…Inverse MatrixLeast SquaresConjugate Transp…Singular value d…Unitary MatrixMatrix RankLinear mapMoore–Penr…

The Moore–Penrose pseudoinverse of a real or complex matrix AA is the unique matrix A+A^+ satisfying four equations that generalize the defining properties of an ordinary matrix inverse. It exists for every finite-dimensional matrix, including rectangular matrices and singular square matrices. Its central role is to produce the minimum-norm solution of a linear least-squares problem, whether the underlying equations have a unique solution, infinitely many solutions, or no exact solution.

Definition

Let A∈Cm×nA\in\mathbb{C}^{m\times n}. Its Moore–Penrose pseudoinverse is the matrix A+∈Cn×mA^+\in\mathbb{C}^{n\times m} satisfying the four Penrose equations:

AA+A=A,AA^+A=A,
A+AA+=A+,A^+AA^+=A^+,
(AA+)∗=AA+,(AA^+)^*=AA^+,
(A+A)∗=A+A.(A^+A)^*=A^+A.

Here M∗M^* denotes the conjugate transpose of MM. For real matrices, it is simply the transpose.

The first two equations express generalized inverse relationships. The last two require the products AA+AA^+ and A+AA^+A to be Hermitian. Together, all four conditions determine A+A^+ uniquely.

The notation A†A^\dagger is also common. The term generalized inverse is broader: a matrix satisfying only AGA=AAGA=A, for example, need not be the Moore–Penrose pseudoinverse and need not be unique.

For a square nonsingular matrix,

A+=A−1.A^+=A^{-1}.

The pseudoinverse of a zero matrix is the zero matrix of the transposed shape.

Construction by singular value decomposition

The singular value decomposition provides a general construction and establishes existence. Write

A=UΣV∗,A=U\Sigma V^*,

where U∈Cm×mU\in\mathbb{C}^{m\times m} and V∈Cn×nV\in\mathbb{C}^{n\times n} are unitary matrices, and Σ\Sigma is rectangular diagonal. If AA has rank rr, its positive singular values are

σ1≥σ2≥⋯≥σr>0.\sigma_1\geq \sigma_2\geq\cdots\geq\sigma_r>0.

Form Σ+\Sigma^+ by transposing the shape of Σ\Sigma, replacing each positive singular value by its reciprocal, and leaving zero entries zero. Then

A+=VΣ+U∗.A^+=V\Sigma^+U^*.

Equivalently, using only the singular vectors associated with positive singular values,

A=∑i=1rσiuivi∗,A+=∑i=1r1σiviui∗.A=\sum_{i=1}^{r}\sigma_i u_i v_i^*, \qquad A^+=\sum_{i=1}^{r}\frac{1}{\sigma_i}v_i u_i^*.

This formula makes the essential operation explicit: the pseudoinverse reverses every nonzero singular direction and assigns zero to directions that cannot be inverted.

Substituting this expression into the four Penrose equations verifies existence. For uniqueness, the Hermitian conditions force AA+AA^+ and A+AA^+A to be specific orthogonal projections; the remaining conditions then fix how A+A^+ acts on their ranges.

Geometric interpretation

Regard AA as a linear map from Cn\mathbb{C}^n to Cm\mathbb{C}^m, equipped with their standard inner products. The domain decomposes as

Cn=ker⁡(A)⊥⊕ker⁡(A),\mathbb{C}^n = \ker(A)^\perp\oplus\ker(A),

where ker⁡(A)\ker(A) is the null space and the superscript ⊥\perp denotes the orthogonal complement. The codomain decomposes as

Cm=range⁡(A)⊕ker⁡(A∗).\mathbb{C}^m = \operatorname{range}(A)\oplus\ker(A^*).

The restriction

A:ker⁡(A)⊥⟶range⁡(A)A:\ker(A)^\perp\longrightarrow\operatorname{range}(A)

is bijective. The pseudoinverse inverts this restricted map and maps ker⁡(A∗)\ker(A^*) to zero.

Consequently,

AA+=Prange⁡(A),A+A=Prange⁡(A∗),AA^+=P_{\operatorname{range}(A)}, \qquad A^+A=P_{\operatorname{range}(A^*)},

where PSP_S is the orthogonal projection onto a subspace SS. The space range⁡(A∗)=ker⁡(A)⊥\operatorname{range}(A^*)=\ker(A)^\perp is the row space, interpreted as a subspace of the domain.

This interpretation explains why the products are not generally identity matrices: directions outside the relevant ranges cannot be recovered. It also shows that the Moore–Penrose construction depends on the inner products. Changing them can change which solution is regarded as minimum-norm.

Linear equations and least squares

Consider a system of linear equations

Ax=b.Ax=b.

For every bb, the vector

x∗=A+bx_*=A^+b

minimizes the Euclidean residual:

x∗∈arg min⁡x∥Ax−b∥2.x_*\in\operatorname*{arg\,min}_{x}\|Ax-b\|_2.

Among all vectors attaining that minimum residual, it is the unique one with the smallest Euclidean norm.

Consistent systems

An exact solution exists precisely when b∈range⁡(A)b\in\operatorname{range}(A), equivalently when

AA+b=b.AA^+b=b.

In that case, every exact solution has the form

x=A+b+(In−A+A)z,z∈Cn,x=A^+b+(I_n-A^+A)z, \qquad z\in\mathbb{C}^n,

where InI_n is the identity matrix. The second term ranges over ker⁡(A)\ker(A).

Since A+bA^+b is orthogonal to the null space, adding any nonzero null-space component increases the norm. Thus the pseudoinverse selects a single solution even when infinitely many exact solutions exist.

Inconsistent systems

If b∉range⁡(A)b\notin\operatorname{range}(A), no exact solution exists. Instead,

Ax∗=AA+bAx_*=AA^+b

is the orthogonal projection of bb onto the column space of AA. The residual

b−AA+bb-AA^+b

is orthogonal to that space.

All least-squares minimizers again have the form

x=A+b+(In−A+A)z.x=A^+b+(I_n-A^+A)z.

They satisfy the normal equations

A∗Ax=A∗b.A^*Ax=A^*b.

When AA is rank-deficient, these equations do not uniquely determine xx; the minimum-norm requirement selects A+bA^+b.

Example

For the single equation

x1+x2=1,A=(11),x_1+x_2=1, \qquad A=\begin{pmatrix}1&1\end{pmatrix},

the pseudoinverse is

A+=(1/21/2).A^+= \begin{pmatrix} 1/2\\ 1/2 \end{pmatrix}.

Hence the minimum-norm solution is

x∗=(1/21/2).x_*= \begin{pmatrix} 1/2\\ 1/2 \end{pmatrix}.

All exact solutions are

x=(1/21/2)+t(1−1),x= \begin{pmatrix} 1/2\\ 1/2 \end{pmatrix} +t \begin{pmatrix} 1\\ -1 \end{pmatrix},

whose squared norms are 1/2+2t21/2+2t^2.

Special formulas and algebraic properties

If AA has full column rank, then A∗AA^*A is invertible and

A+=(A∗A)−1A∗.A^+=(A^*A)^{-1}A^*.

If AA has full row rank, then AA∗AA^* is invertible and

A+=A∗(AA∗)−1.A^+=A^*(AA^*)^{-1}.

Neither formula with an ordinary inverse applies without its rank assumption. General identities using pseudoinverses are

A+=(A∗A)+A∗=A∗(AA∗)+.A^+=(A^*A)^+A^* =A^*(AA^*)^+.

Other useful properties include

(A+)+=A,(A∗)+=(A+)∗,(A^+)^+=A, \qquad (A^*)^+=(A^+)^*,
(αA)+=α−1A+for α≠0,(\alpha A)^+=\alpha^{-1}A^+ \quad\text{for }\alpha\neq0,

and

rank⁡(A+)=rank⁡(A).\operatorname{rank}(A^+)=\operatorname{rank}(A).

For compatible unitary matrices UU and VV,

(UAV)+=V∗A+U∗.(UAV)^+=V^*A^+U^*.

Thus the construction behaves naturally under orthonormal changes of coordinates. It does not generally obey the reverse-order law

(AB)+=B+A+;(AB)^+=B^+A^+;

that identity requires additional conditions. Nor does the pseudoinverse generally distribute over matrix addition.

For a Hermitian matrix, the spectral theorem gives an analogous construction: retain its eigenvectors, replace each nonzero eigenvalue by its reciprocal, and leave zero eigenvalues zero.

Numerical computation and sensitivity

In numerical linear algebra, the SVD is a standard method for computing pseudoinverses and minimum-norm solutions. Rank-revealing forms of QR decomposition and related factorizations are also used.

If the aim is only to obtain A+bA^+b, explicitly forming the entire matrix A+A^+ is often unnecessary. A factorization can be applied directly to bb, reducing storage and sometimes computational cost.

The full-column-rank expression (A∗A)−1A∗(A^*A)^{-1}A^* is useful algebraically but can be numerically undesirable. Forming A∗AA^*A squares the spectral condition number:

κ2(A∗A)=κ2(A)2.\kappa_2(A^*A)=\kappa_2(A)^2.

This can magnify the effects of rounding errors.

Small singular values and numerical rank

In exact mathematics, every nonzero singular value is inverted. In floating-point arithmetic, extremely small singular values may be indistinguishable from perturbations or rounding error. Software therefore commonly uses a tolerance and treats singular values below it as zero.

The resulting matrix is the exact pseudoinverse of a truncated approximation to AA, rather than necessarily the exact pseudoinverse of AA. The tolerance determines the effective numerical rank.

Even when a small singular value is accurately computed, its reciprocal can amplify measurement noise. Numerical stability of the factorization does not eliminate sensitivity inherent in the problem.

Continuity and rank changes

The pseudoinverse varies continuously along perturbations that preserve rank, but continuity can fail when rank changes. For example,

A(t)=(100t).A(t)= \begin{pmatrix} 1&0\\ 0&t \end{pmatrix}.

For t≠0t\neq0,

A(t)+=(1001/t),A(t)^+= \begin{pmatrix} 1&0\\ 0&1/t \end{pmatrix},

whereas

A(0)+=(1000).A(0)^+= \begin{pmatrix} 1&0\\ 0&0 \end{pmatrix}.

The divergence of 1/t1/t illustrates why nearly singular problems can be highly sensitive.

Regularization and applications

The pseudoinverse is central to linear regression, constrained linear calculations, and inverse problems. In ordinary least squares, A+bA^+b gives a minimum-norm coefficient vector even when the design matrix is rank-deficient. This algebraic choice does not, by itself, make nonidentifiable coefficients uniquely interpretable.

In inverse problems, direct pseudoinversion may amplify noise. Regularization replaces exact inversion with a modified problem. For example, minimizing

∥Ax−b∥22+λ∥x∥22,λ>0,\|Ax-b\|_2^2+\lambda\|x\|_2^2, \qquad \lambda>0,

gives

xλ=(A∗A+λIn)−1A∗b.x_\lambda=(A^*A+\lambda I_n)^{-1}A^*b.

This is the basic quadratic penalty used in ridge regression. In singular-vector coordinates, it replaces the reciprocal factor 1/σi1/\sigma_i with

σiσi2+λ.\frac{\sigma_i}{\sigma_i^2+\lambda}.

For a fixed finite-dimensional matrix,

A+=lim⁡λ↓0(A∗A+λIn)−1A∗.A^+ = \lim_{\lambda\downarrow0} (A^*A+\lambda I_n)^{-1}A^*.

Truncating small singular values is another regularization strategy. Both approaches modify the treatment of weakly determined directions; neither should be confused with exact pseudoinversion of the original matrix.

History and scope

The construction is named after Eliakim Hastings Moore, who introduced a generalized inverse in 1920, and Roger Penrose, who gave the four-equation characterization in 1955. It became a standard tool connecting linear algebra, least squares, and numerical computation.

The finite-dimensional theory extends to operators between Hilbert spaces. For bounded operators with closed range, the Moore–Penrose inverse is a bounded operator with analogous projection properties. Without closed range, a generalized inverse can be unbounded and need not be defined on the entire codomain, so the finite-dimensional existence statement requires qualification in infinite dimensions.