Bias-variance analysis is a useful finite-dimensional identity, but in learning theory it is only one piece of the larger approximation-estimation tradeoff.
Squared-error decomposition
For regression estimators, the mean squared error splits into noise, squared bias, and variance.
Theorem. Let \(Y=m(X)+\varepsilon\), \(\mathbb E(\varepsilon\mid X)=0\), and let \(\hat m(x)\) be a random estimator trained on data independent of a fresh \((X,Y)\). Then \[ \mathbb E\{Y-\hat m(X)\}^2 =\mathbb E\sigma^2(X) +\mathbb E\{\mathbb E\hat m(X)-m(X)\}^2 +\mathbb E\{\hat m(X)-\mathbb E\hat m(X)\}^2, \] where \(\sigma^2(X)=\mathbb E(\varepsilon^2\mid X)\).
Proof. Condition on \(X\). Write \[ Y-\hat m(X)=\varepsilon+m(X)-\mathbb E\hat m(X)+\mathbb E\hat m(X)-\hat m(X). \] The three components have zero cross terms after taking expectation: the noise has conditional mean zero, and the centered estimator has mean zero over the training randomness. The identity follows.
Approximation and estimation
In function-class learning, approximation error is deterministic and estimation error is stochastic.
Definition. For a class \(\mathcal F_k\) indexed by complexity \(k\), define \[ A(k)=\inf_{f\in\mathcal F_k}R(f)-R(f^*), \qquad E_n(k)=R(\hat f_{k,n})-\inf_{f\in\mathcal F_k}R(f). \] The risk is controlled by \(A(k)+E_n(k)\).
Proposition. If \(A(k)\lesssim k^{-2s/d}\) and \(E_n(k)\lesssim k/n\), then the optimized rate is \[ \inf_k\{A(k)+E_n(k)\}\lesssim n^{-2s/(2s+d)}. \]
Proof. Balance \(k^{-2s/d}\) with \(k/n\). The equality \(k^{-2s/d}\asymp k/n\) gives \(k\asymp n^{d/(2s+d)}\). Substituting gives \(n^{-2s/(2s+d)}\).
Smoothing parameters
A smoothing parameter controls bias by restricting roughness and variance by averaging observations.
Example. For a kernel smoother with bandwidth \(h\) in \(d\) dimensions, a typical heuristic risk is \[ h^{2s}+\frac{1}{nh^d}. \] The first term is squared bias for \(s\)-smooth functions; the second is variance from an effective sample size \(nh^d\).
Proposition. Optimizing \(h^{2s}+1/(nh^d)\) yields \(h\asymp n^{-1/(2s+d)}\) and risk \(n^{-2s/(2s+d)}\).
Proof. Set \(h^{2s}\asymp 1/(nh^d)\), hence \(h^{2s+d}\asymp n^{-1}\). The claimed bandwidth and risk follow.
Beyond the classical identity
Modern learning often has interpolation, implicit regularization, and nonconvex optimization. Bias-variance language remains useful only when tied back to risk.
Remark. A method can have zero training error and still generalize when the algorithmic bias selects a low-complexity interpolating solution. In such settings, norm, margin, stability, or compression may be more informative than the classical parameter-count variance.
Orthogonal projection in regression
The regression decomposition is an instance of Hilbert-space projection. Let \(\mathcal H=L^2(P_X)\) and let \(\mathcal V\subset\mathcal H\) be a closed linear subspace.
Theorem. The best approximation \(m_{\mathcal V}=\operatorname{proj}_{\mathcal V}m\) satisfies \[ \|m-f\|_{L^2(P_X)}^2 =\|m-m_{\mathcal V}\|_2^2+\|m_{\mathcal V}-f\|_2^2, \qquad f\in\mathcal V. \] Therefore, for any random \(\widehat m\in\mathcal V\), \[ \mathbb E\|\widehat m-m\|_2^2 =\|m-m_{\mathcal V}\|_2^2+ \mathbb E\|\widehat m-m_{\mathcal V}\|_2^2. \]
Proof. The projection residual \(m-m_{\mathcal V}\) is orthogonal to \(\mathcal V\). Since \(m_{\mathcal V}-f\in\mathcal V\), the cross term in the squared norm vanishes. Apply the identity pointwise in the training sample with \(f=\widehat m\), then average.
Exact risk of ordinary least squares
Theorem. In the fixed-design model \(Y=X\beta+\varepsilon\), with \(X\in\mathbb R^{n\times p}\) of full column rank, \(\mathbb E\varepsilon=0\), and \(\operatorname{Cov}(\varepsilon)=\sigma^2I_n\), ordinary least squares obeys \[ \mathbb E\|\widehat\beta-\beta\|_2^2 =\sigma^2\operatorname{tr}\{(X^\top X)^{-1}\}, \] and its in-sample prediction risk is \[ \frac1n\mathbb E\|X\widehat\beta-X\beta\|_2^2 =\frac{\sigma^2p}{n}. \]
Proof. Since \(\widehat\beta-\beta=(X^\top X)^{-1}X^\top\varepsilon\), its covariance is \(\sigma^2(X^\top X)^{-1}\); trace gives the first identity. For prediction, \(X(\widehat\beta-\beta)=H\varepsilon\), where \(H=X(X^\top X)^{-1}X^\top\) is an orthogonal projector. Thus \(\mathbb E\|H\varepsilon\|_2^2=\sigma^2\operatorname{tr}(H)=\sigma^2p\).
Ridge shrinkage in the singular basis
Proposition. Let \(X=U\operatorname{diag}(s_1,\ldots,s_r)V^\top\) be a singular-value decomposition. The ridge estimator \(\widehat\beta_\lambda=(X^\top X+\lambda I)^{-1}X^\top Y\) has, in the \(V\)-basis, \[ \mathbb E(v_j^\top\widehat\beta_\lambda) =\frac{s_j^2}{s_j^2+\lambda}\,v_j^\top\beta, \qquad \operatorname{Var}(v_j^\top\widehat\beta_\lambda) =\frac{\sigma^2s_j^2}{(s_j^2+\lambda)^2}. \]
Proof. Substitute the SVD into the ridge formula: \[ \widehat\beta_\lambda =V\operatorname{diag}\!\left(\frac{s_j}{s_j^2+\lambda}\right)U^\top Y. \] Insert \(Y=U\operatorname{diag}(s_j)V^\top\beta+\varepsilon\), then take coordinatewise means and variances.
Small-singular-value directions are precisely where least squares has high variance. Ridge suppresses those directions most strongly, accepting bias in exchange for a potentially much larger variance reduction.
Nonparametric series estimators
Example. In Gaussian white noise, \[ Y_j=\theta_j+\frac{\sigma}{\sqrt n}\xi_j,\qquad \xi_j\stackrel{\mathrm{iid}}{\sim}N(0,1), \] consider the truncation estimator \(\widehat\theta_j=Y_j\mathbf1(j\le k)\). Its exact squared \(\ell^2\)-risk is \[ \mathbb E_\theta\|\widehat\theta-\theta\|_2^2 =\sum_{j>k}\theta_j^2+\frac{\sigma^2k}{n}. \] On the Sobolev ellipsoid \(\sum_{j\ge1}j^{2s}\theta_j^2\le L^2\), the bias is at most \(L^2k^{-2s}\). Balancing it against \(k/n\) gives \(k\asymp n^{1/(2s+1)}\) and risk \(n^{-2s/(2s+1)}\).
Model selection adds search variance
Remark. A bias-variance calculation conditional on a selected model omits the randomness of selection. If one searches over \(M\) candidate models, the effective stochastic price generally contains \(\log M\), not merely the dimension of the winner. Oracle inequalities account for this search cost; a post-selection calculation that treats the selected model as fixed does not.
Interpolation and the minimum-norm solution
Proposition. If \(p>n\) and \(X\) has full row rank, the minimum-Euclidean-norm interpolator is \[ \widehat\beta_{\min}=X^\top(XX^\top)^{-1}Y. \] Its prediction at \(x\) depends only on the projection of \(x\) onto the row space of \(X\); components in \(\ker(X)\) are set to zero.
Proof. Every interpolator has the form \(X^\top(XX^\top)^{-1}Y+v\) with \(v\in\ker(X)\). The first summand lies in \(\operatorname{row}(X)=\ker(X)^\perp\), so Pythagoras gives \(\|\beta\|_2^2=\|\widehat\beta_{\min}\|_2^2+\|v\|_2^2\). Hence \(v=0\) uniquely minimizes norm.
The minimum-norm constraint is an algorithmic inductive bias. Whether interpolation generalizes depends on the alignment of the signal and covariance spectrum, not on training error alone. This spectral viewpoint reappears in kernel ridgeless regression and neural tangent models.