Hilbert spaces retain enough Euclidean geometry to make orthogonality, projection, and least squares exact. Completeness turns the geometric minimization picture into a theorem.
Inner products
An inner product gives length and angle, so Hilbert-space arguments can use geometry rather than only norm estimates.
Definition. A Hilbert space is a complete inner-product space \(H\). We take \(\langle\cdot,\cdot\rangle\) linear in the first variable and conjugate-linear in the second. The norm is \(\|x\|=\langle x,x\rangle^{1/2}\).
Theorem. Cauchy-Schwarz inequality: \(|\langle x,y\rangle|\le\|x\|\|y\|\) for all \(x,y\in H\).
Proof. If \(y=0\) there is nothing to prove. Otherwise, for any \(\lambda\in\mathbb F\), \(0\le\|x-\lambda y\|^2=\|x\|^2-2\operatorname{Re}(\lambda\langle y,x\rangle)+|\lambda|^2\|y\|^2\). Choose \(\lambda=\langle x,y\rangle/\|y\|^2\) with the convention matching the inner-product linearity. The resulting inequality is \(\|x\|^2-|\langle x,y\rangle|^2/\|y\|^2\ge0\).
Parallelogram and polarization
The parallelogram identity characterizes norms that come from inner products. Polarization then recovers the hidden inner product from the norm.
Proposition. The norm of a Hilbert space satisfies \(\|x+y\|^2+\|x-y\|^2=2\|x\|^2+2\|y\|^2\). Conversely, a norm satisfying the parallelogram identity comes from an inner product by the polarization identity.
Proof. Expanding \(\langle x\pm y,x\pm y\rangle\) gives the parallelogram identity. Conversely, define the inner product by the real or complex polarization formula. The parallelogram identity is exactly the algebra needed to prove additivity; homogeneity and positivity follow from norm properties.
Projection theorem
Projection is the Hilbert-space form of best approximation. Closed convex sets have nearest points because orthogonality supplies a variational inequality.
Theorem. Let \(C\subset H\) be nonempty, closed, and convex. For every \(x\in H\), there is a unique \(p=P_Cx\in C\) minimizing \(\|x-c\|\) over \(c\in C\). It is characterized by \(\operatorname{Re}\langle x-p,c-p\rangle\le0\) for all \(c\in C\).
Proof. Let \(d=\inf_{c\in C}\|x-c\|\) and choose \(c_n\in C\) with \(\|x-c_n\|\to d\). Since \(C\) is convex, \((c_n+c_m)/2\in C\). The parallelogram identity applied to \(x-c_n\) and \(x-c_m\) gives \(\|c_n-c_m\|^2\le2\|x-c_n\|^2+2\|x-c_m\|^2-4d^2\), which tends to \(0\). Thus \(c_n\) is Cauchy, converges to some \(p\in C\), and \(\|x-p\|=d\).
If \(p,q\) were two minimizers, the same parallelogram argument with their midpoint gives \(\|p-q\|=0\). For the variational inequality, minimize \(\|x-(p+t(c-p))\|^2\) over \(0\le t\le1\) and take the right derivative at \(0\). Conversely, that inequality implies \(\|x-c\|^2=\|x-p\|^2+\|c-p\|^2-2\operatorname{Re}\langle x-p,c-p\rangle\ge\|x-p\|^2\).
Closed subspaces
Closed subspaces preserve the ambient analytic structure. They are the subspaces for which limits, quotients, and orthogonal or weak arguments behave correctly.
Corollary. If \(M\subset H\) is a closed subspace, then \(H=M\oplus M^\perp\). The map \(P_Mx\) is linear, bounded, self-adjoint, idempotent, and \(\|P_M\|\le1\).
Proof. For \(p=P_Mx\), the variational inequality applied to \(p\pm m\) and \(p\pm im\) gives \(\langle x-p,m\rangle=0\) for all \(m\in M\). Thus \(x=p+(x-p)\in M+M^\perp\). The intersection is \(\{0\}\). Linearity follows from uniqueness of the decomposition. Boundedness follows from \(\|P_Mx\|\le\|x\|\), and self-adjointness follows from \(\langle P_Mx,y\rangle=\langle P_Mx,P_My\rangle=\langle x,P_My\rangle\).
Normal equations
The normal equations express least squares as orthogonality of the residual. This is the geometric core of projection methods and linear regression.
Example. Least squares is projection. If \(A:\mathbb F^p\to H\) has closed range \(M\), the best approximation to \(y\in H\) in \(M\) is \(A\hat\beta=P_My\), and the residual satisfies \(A^*(y-A\hat\beta)=0\). In matrix language this is \(A^*A\hat\beta=A^*y\), the normal equation.