The singular value decomposition works for every matrix, square or rectangular, diagonalizable or not. It describes how a linear map stretches orthogonal input directions into orthogonal output directions.
Singular values and singular vectors
Definition. For \(A:F^n\to F^m\), the singular values of \(A\) are the nonnegative square roots of the eigenvalues of \(A^*A\). A right singular vector \(v_i\) and left singular vector \(u_i\) satisfy \[ Av_i=\sigma_i u_i, \qquad A^*u_i=\sigma_i v_i. \]
The singular value decomposition
Theorem. Every \(m\times n\) matrix \(A\) over \(\mathbb R\) or \(\mathbb C\) has a decomposition \[ A=U\Sigma V^*, \] where \(U\) and \(V\) are unitary or orthogonal, and \(\Sigma\) is diagonal rectangular with entries \(\sigma_1\ge\cdots\ge\sigma_r\gt0\) followed by zeros.
Proof. Apply the spectral theorem to \(A^*A\), obtaining an orthonormal eigenbasis \(v_i\) with eigenvalues \(\sigma_i^2\). For \(\sigma_i\gt0\), set \(u_i=Av_i/\sigma_i\). These \(u_i\)'s are orthonormal because \(\langle Av_i,Av_j\rangle=\langle v_i,A^*Av_j\rangle=\sigma_j^2\delta_{ij}\). Extend the \(u_i\)'s to an orthonormal basis of \(F^m\). The matrix of \(A\) in these bases is \(\Sigma\).
Geometric meaning
Proposition. The map \(A\) sends the unit sphere in \(F^n\) to an ellipsoid in \(F^m\) whose semiaxis lengths are the singular values \(\sigma_i\), in the directions \(u_i\). The largest singular value is \[ \|A\|_2=\max_{\|x\|=1}\|Ax\|=\sigma_1. \]
A rectangular SVD by inspection
Example. Let \[ A=\begin{pmatrix}3&0\\0&1\\0&0\end{pmatrix}:F^2\to F^3. \] The standard bases already give an SVD: the singular values are \(3\) and \(1\), the right singular vectors are \(e_1,e_2\), and the left singular vectors are \(e_1,e_2\) in \(F^3\), with \(e_3\) completing the left orthonormal basis. The map stretches one orthogonal direction by \(3\), another by \(1\), and embeds the result in a plane.
Calculation. Here \[ A^*A=\begin{pmatrix}9&0\\0&1\end{pmatrix}, \] so the singular values are the square roots of the eigenvalues of \(A^*A\). This calculation is often the quickest way to find the right singular vectors first.
Rank, kernel, and range
Corollary. The rank of \(A\) is the number of positive singular values. The kernel is spanned by right singular vectors with \( \sigma_i=0\), and the range is spanned by left singular vectors with \(\sigma_i\gt0\).
Pseudoinverse
Definition. If \(A=U\Sigma V^*\), the Moore-Penrose pseudoinverse is \[ A^+=V\Sigma^+U^*, \] where \(\Sigma^+\) replaces each positive \(\sigma_i\) by \(1/\sigma_i\) and transposes the rectangular shape. The vector \(A^+b\) is the minimum-norm least-squares solution of \(Ax\approx b\).
Best low-rank approximation
Theorem. If \(A=\sum_i\sigma_i u_iv_i^*\) is an SVD with \(\sigma_1\ge\cdots\), then \[ A_k=\sum_{i=1}^k\sigma_i u_iv_i^* \] is a best rank-\(k\) approximation to \(A\) in spectral norm and Frobenius norm. Moreover, \(\|A-A_k\|_2=\sigma_{k+1}\) and \(\|A-A_k\|_F^2=\sum_{i>k}\sigma_i^2\).
Remark. The SVD is the most reliable coordinate system for a linear map because it uses orthonormal bases on both sides. It reveals rank, conditioning, least squares behavior, and low-dimensional structure at once.
Existence and geometry of the SVD
Theorem. Every \(A\in\mathbb C^{m\times n}\) has a factorization \[ A=U\Sigma V^*, \] where \(U,V\) are unitary and \(\Sigma\) is rectangular diagonal with \(\sigma_1\ge\cdots\ge\sigma_r>0\). The nonzero \(\sigma_i^2\) are the eigenvalues of \(A^*A\).
Proof. The spectral theorem gives an orthonormal eigenbasis \(v_i\) of the positive semidefinite matrix \(A^*A\), with eigenvalues \(\sigma_i^2\). For \(\sigma_i>0\), set \(u_i=Av_i/\sigma_i\). Then \[ \langle u_i,u_j\rangle =\frac{\langle v_i,A^*Av_j\rangle}{\sigma_i\sigma_j} =\delta_{ij}. \] Complete the \(u_i\)'s and \(v_i\)'s to orthonormal bases. The identities \(Av_i=\sigma_iu_i\) and \(Av_i=0\) on \(\ker A\) give the factorization.
The unit sphere first rotates by \(V^*\), stretches independently by the singular values, and rotates by \(U\). Hence its image is an ellipsoid with principal semiaxes \(\sigma_i\).
Pseudoinverse and least squares
Define \[ A^+=V\Sigma^+U^*, \] where each nonzero \(\sigma_i\) is replaced by \(\sigma_i^{-1}\). Then \(AA^+\) is the orthogonal projection onto \(\operatorname{range}A\), and \(A^+A\) projects onto \((\ker A)^\perp\). Consequently \(A^+b\) is the unique least-squares solution of minimum norm.
Optimal low-rank approximation
Theorem (Eckart-Young-Mirsky). If \[ A_k=\sum_{i=1}^k\sigma_iu_iv_i^*, \] then among matrices \(B\) of rank at most \(k\), \[ \|A-B\|_2\ge\sigma_{k+1},\qquad \|A-B\|_F^2\ge\sum_{i>k}\sigma_i^2, \] with equality for \(B=A_k\).
Proof. Since \(\dim\operatorname{span}(v_1,\ldots,v_{k+1})=k+1\), a rank-\(k\) map \(B\) annihilates some unit vector \(x\) in this span. Therefore \[ \|A-B\|_2\ge\|(A-B)x\|=\|Ax\|\ge\sigma_{k+1}. \] Unitary invariance and orthogonality of matrix entries give the Frobenius bound by retaining at most \(k\) diagonal singular components.