Orthogonal projection turns geometry into an optimization principle: the best approximation is characterized by an error vector perpendicular to the approximating subspace.
The projection theorem
Theorem. Let \(U\) be a finite-dimensional subspace of an inner product space \(V\). For every \(v\in V\), there is a unique decomposition \[ v=u+r, \qquad u\in U, \qquad r\in U^\perp. \] The vector \(u\) is the unique element of \(U\) minimizing \(\|v-w\|\) over \(w\in U\).
Proof. Choose an orthonormal basis \(q_1, \ldots,q_k\) of \(U\) and set \(u=\sum_j\langle v,q_j\rangle q_j\). Then \(v-u\) is orthogonal to each \(q_j\), hence to \(U\). If \(w\in U\), then \(v-w=(v-u)+(u-w)\) with orthogonal summands, so \[ \|v-w\|^2=\|v-u\|^2+\|u-w\|^2, \] minimized uniquely at \(w=u\).
Projection matrices
Proposition. If the columns of \(Q\in F^{m\times k}\) are orthonormal, then the orthogonal projection onto \(\operatorname{col}Q\) is \[ P=QQ^*. \] The matrix \(P\) satisfies \(P^2=P\) and \(P^*=P\). Conversely, any self-adjoint idempotent matrix is an orthogonal projection.
Least squares and normal equations
Theorem. For \(A\in F^{m\times n}\) and \(b\in F^m\), a vector \(\hat x\) minimizes \(\|Ax-b\|\) if and only if \[ A^*(A\hat x-b)=0, \] equivalently \[ A^*A\hat x=A^*b. \] These are the normal equations.
Proof. The set \(\{Ax:x\in F^n\}\) is the column space of \(A\). The closest point \(A\hat x\) to \(b\) is the projection of \(b\) onto that column space. The residual \(b-A\hat x\) must therefore be orthogonal to every column of \(A\), which is exactly \(A^*(b-A\hat x)=0\).
Full column rank formula
Corollary. If \(A\) has full column rank, then \(A^*A\) is invertible and the least-squares solution is unique: \[ \hat x=(A^*A)^{-1}A^*b. \] The projection matrix onto \(\operatorname{col}A\) is \(P=A(A^*A)^{-1}A^*\).
QR factorization
Proposition. If \(A\in F^{m\times n}\) has linearly independent columns, Gram-Schmidt gives \(A=QR\), where \(Q\) has orthonormal columns and \(R\) is upper triangular with nonzero diagonal entries. Then least squares reduces to \[ R\hat x=Q^*b. \]
Example. Fitting a line \(y=\alpha+ \beta t\) to data points \((t_i,y_i)\) is the least-squares problem \(Ax\approx y\) with rows \((1,t_i)\). The residual vector is orthogonal to the columns \((1, \ldots,1)^\top\) and \((t_1, \ldots,t_m)^\top\), giving the classical normal equations for linear regression.
Projection theorem and normal equations
Projection theorem. For every subspace \(S\) of a finite-dimensional inner-product space and every \(y\), there is a unique decomposition \[ y=s+r,\qquad s\in S,\quad r\in S^\perp. \] The vector \(s=P_Sy\) uniquely minimizes \(\|y-z\|\) over \(z\in S\).
Proof. Choose an orthonormal basis \((q_i)\) of \(S\) and set \[ s=\sum_i\langle y,q_i\rangle q_i. \] Then \(r=y-s\) is orthogonal to \(S\). For \(z\in S\), \[ \|y-z\|^2=\|r\|^2+\|s-z\|^2, \] so the minimum is attained uniquely at \(z=s\).
For a matrix \(X\), least squares minimizes \(\|y-X\beta\|_2^2\). The residual must be orthogonal to \(\operatorname{range}X\), giving \[ X^*(y-X\widehat\beta)=0,\qquad X^*X\widehat\beta=X^*y. \] If \(X\) has full column rank, \[ \widehat\beta=(X^*X)^{-1}X^*y,\qquad P_X=X(X^*X)^{-1}X^*. \] For deficient rank, \(X^+y\) is the minimum-norm solution and \(P_X=XX^+\).
Why QR is preferable to normal equations
If \(X=QR\) has orthonormal columns in \(Q\), solve \[ R\widehat\beta=Q^*y. \] Forming \(X^*X\) squares the spectral condition number: \[ \kappa_2(X^*X)=\kappa_2(X)^2. \] The projection geometry is exact in either derivation, but QR or SVD better preserves it numerically.