Approximation theory asks for the best finite representation of a function under a chosen norm or inner product. Orthogonality turns this into geometry.
Best approximation
Definition. For a normed space \(X\) and finite-dimensional subspace \(V_n\), a best approximation to \(f\) is an element \(v_n\in V_n\) minimizing \(\|f-v\|\) over \(v\in V_n\). The answer depends strongly on the norm.
Projection theorem
Theorem. In a Hilbert space, the best approximation from a closed subspace \(V\) is the orthogonal projection \(P_Vf\), characterized by \(\langle f-P_Vf,v\rangle=0\) for every \(v\in V\). This is the prototype of Galerkin methods.
Orthogonal polynomial expansion
Proposition. If \(\{p_k\}\) is orthogonal for weight \(w\) on \([a,b]\), then the least-squares polynomial approximation is obtained by truncating the series of coefficients \(\langle f,p_k\rangle_w/\langle p_k,p_k\rangle_w\).
Chebyshev approximation
Remark. Chebyshev polynomials organize near-minimax approximation on an interval. Their equioscillation behavior explains why they appear in interpolation nodes, spectral methods, and stable polynomial evaluation.
Approximation versus computation
Remark. A good approximation space is not enough; the basis must also be numerically usable. Orthogonal or nearly orthogonal bases usually behave better than monomial bases because coefficient perturbations are less amplified.
Orthogonal projection
Let \(w(x)>0\) on \([a,b]\) and \[ \langle f,g\rangle_w=\int_a^bf(x)g(x)w(x)\,\mathrm{d}x. \] Gram--Schmidt produces orthogonal polynomials \(\phi_0,\phi_1,\ldots\), with \(\deg\phi_n=n\).
Theorem (best \(L_2\) approximation). The polynomial \[ P_nf=\sum_{k=0}^n \frac{\langle f,\phi_k\rangle_w}{\|\phi_k\|_w^2}\phi_k \] uniquely minimizes \(\|f-p\|_w\) over \(p\in\mathcal P_n\), and \[ \|f-P_nf\|_w^2 =\|f\|_w^2- \sum_{k=0}^n \frac{|\langle f,\phi_k\rangle_w|^2}{\|\phi_k\|_w^2}. \]
Proof. \(f-P_nf\) is orthogonal to \(\mathcal P_n\). For any \(p\in\mathcal P_n\), \[ \|f-p\|_w^2 =\|f-P_nf\|_w^2+\|P_nf-p\|_w^2. \] This proves minimality, uniqueness, and the coefficient identity.
Three-term recurrence
Theorem. Monic orthogonal polynomials satisfy \[ \phi_{n+1}(x)=(x-\alpha_n)\phi_n(x)-\beta_n\phi_{n-1}(x), \qquad \beta_n>0. \]
Proof. Expand \(x\phi_n\) in the orthogonal basis through degree \(n+1\). For \(k\le n-2\), \(\langle x\phi_n,\phi_k\rangle =\langle\phi_n,x\phi_k\rangle=0\), since \(x\phi_k\) has degree at most \(n-1\). Only three coefficients remain. Comparing leading terms and norms gives the form and \(\beta_n>0\).
The recurrence gives stable \(O(n)\) evaluation through the Clenshaw algorithm and converts Gaussian quadrature nodes into eigenvalues of a symmetric tridiagonal Jacobi matrix.
Chebyshev approximation
For \(w(x)=(1-x^2)^{-1/2}\), the orthogonal polynomials are \(T_k(x)=\cos(k\arccos x)\). If \(f\) extends analytically to the Bernstein ellipse \(E_\rho\), \(\rho>1\), and \(|f|\le M\) there, then Chebyshev coefficients satisfy \[ |a_k|\le2M\rho^{-k}, \] so truncation converges geometrically. Finite smoothness instead yields algebraic coefficient decay determined by derivatives.
Uniform best approximation
The minimax polynomial in sup norm is characterized by the alternation theorem: \(p^*\in\mathcal P_n\) is best if and only if \(f-p^*\) attains alternating extrema of equal magnitude at least \(n+2\) ordered points. The Remez algorithm exploits this characterization. Orthogonal projection and minimax approximation solve different normed problems.