Infinite-dimensional estimation requires a mechanism that prevents the empirical criterion from fitting arbitrarily fine noise. A sieve imposes a hard complexity bound; a penalty imposes a soft one. In both cases, approximation error decreases as the admissible model expands, while empirical-process fluctuation increases.
Sieve estimators
Definition. A sieve is a sequence \(\Theta_n\subset\Theta\), usually finite dimensional, whose union is dense in the target space. A sieve \(M\)-estimator satisfies \[ M_n(\hat\theta_n) \geq\sup_{\theta\in\Theta_n}M_n(\theta)-o_p(1). \] Let \[ \theta_n^\circ\in\arg\max_{\theta\in\Theta_n}M(\theta) \] be a population sieve projection.
Proposition (basic sieve decomposition). For any metric \(d\), \[ d(\hat\theta_n,\theta_0) \leq d(\hat\theta_n,\theta_n^\circ) +d(\theta_n^\circ,\theta_0). \] The second term is approximation bias. The first is stochastic estimation error within \(\Theta_n\), controlled by local curvature and entropy of the sieve criterion class.
The decomposition is elementary but organizes the proof: first approximate the target by a sieve element, then analyze the empirical optimizer relative to that projection. Treating \(\theta_n^\circ\) as if it were exactly \(\theta_0\) can conceal a bias larger than the stochastic limit.
Series least squares
Theorem (rate calculation). Suppose \(g_0\) is \(s\)-smooth on \([0,1]^d\), and a \(k\)-dimensional basis admits an approximation \(g_k^\circ\) satisfying \[ \|g_k^\circ-g_0\|_{P_X,2}\lesssim k^{-s/d}. \] If the empirical Gram matrix is well conditioned and noise has bounded conditional variance, series least squares has \[ \|\hat g_k-g_0\|_{P_X,2} = O_p\!\left(k^{-s/d}+\sqrt{k/n}\right). \] Balancing the two terms gives \[ k\asymp n^{d/(2s+d)}, \qquad \|\hat g_k-g_0\|_{P_X,2} =O_p(n^{-s/(2s+d)}). \]
Proof outline. Orthogonality of least squares gives a basic inequality comparing \(\hat g_k\) with \(g_k^\circ\). The quadratic term is the empirical \(L_2\)-norm of their difference; the noise term is an empirical process over a \(k\)-dimensional linear ball. Euclidean entropy or direct linear algebra bounds that term by \(O_p(\sqrt{k/n})\) in \(L_2\)-norm. Uniform Gram-matrix control transfers empirical norm to population norm. Add the approximation error and optimize \(k\). \(\square\)
Penalized empirical risk
Definition. Given a nonnegative complexity functional \(J\), a penalized estimator minimizes \[ P_n\ell_\theta+\lambda_nJ(\theta) \] or equivalently maximizes \(P_nm_\theta-\lambda_nJ(\theta)\).
The penalty is not only a numerical device. Its sublevel sets \[ \Theta(R)=\{\theta:J(\theta)\leq R\} \] often have finite entropy. The basic inequality shows that \(\hat\theta_n\) lies in a random sublevel set whose radius can be bounded, thereby localizing the empirical process.
The penalized least-squares basic inequality
Theorem. In the regression model \(Y=f_0(X)+\varepsilon\), \(E(\varepsilon\mid X)=0\), let \[ \hat f\in\arg\min_{f\in\mathcal F} \left\{P_n(Y-f(X))^2+\lambda J(f)\right\}. \] Then, for every \(f\in\mathcal F\), \[ \|\hat f-f_0\|_{P_n,2}^2 +\lambda J(\hat f) \leq \|f-f_0\|_{P_n,2}^2 +\lambda J(f) +2P_n\{\varepsilon(\hat f-f)\}. \]
Proof. Optimality gives \[ P_n(Y-\hat f)^2+\lambda J(\hat f) \leq P_n(Y-f)^2+\lambda J(f). \] Substitute \(Y=f_0+\varepsilon\) and expand both squares. The common \(P_n\varepsilon^2\) terms cancel, leaving the displayed inequality. \(\square\)
All stochastic analysis is now concentrated in the multiplier-type term \(P_n\{\varepsilon(\hat f-f)\}\). One bounds it uniformly over sets indexed by both prediction distance and penalty radius, then absorbs part of the quadratic loss into the left side.
An abstract oracle inequality
Proposition. Suppose an event of probability at least \(1-\eta\) satisfies, for every \(g\) in the relevant difference class, \[ 2|P_n(\varepsilon g)| \leq\frac12\|g\|_{P_n,2}^2+\operatorname{pen}(g). \] Then the basic inequality implies, on this event, \[ \|\hat f-f_0\|_{P_n,2}^2 \lesssim \inf_{f\in\mathcal F} \left[ \|f-f_0\|_{P_n,2}^2 +\lambda J(f) +\operatorname{pen}(\hat f-f) \right]. \]
A useful oracle inequality makes the last penalty deterministic or bounds it by the same complexity assigned to \(f\). The result then compares \(\hat f\) with every candidate, automatically selecting a near-optimal approximation–estimation tradeoff.
Sparsity penalties
Example (lasso basic inequality). In fixed-design linear regression \(Y=X\beta_0+\varepsilon\), \[ \hat\beta \in\arg\min_\beta \left\{\frac1n\|Y-X\beta\|_2^2+2\lambda\|\beta\|_1\right\}. \] With \(\Delta=\hat\beta-\beta_0\), \[ \frac1n\|X\Delta\|_2^2 2\lambda\|\hat\beta\|_1 \leq \frac2n\varepsilon^\top X\Delta 2\lambda\|\beta_0\|_1. \] On \(\|X^\top\varepsilon/n\|_\infty\leq\lambda/2\), duality bounds the stochastic term by \(\lambda\|\Delta\|_1\). Decomposability of the \(\ell^1\)-norm yields a cone condition, and a restricted eigenvalue converts prediction error to coefficient error.
Quadratic penalties and effective dimension
Example (spectral regularization). For a Hilbert-space penalty \(J(f)=\|f\|_{\mathcal H}^2\), diagonalize the empirical or population covariance operator with eigenvalues \(\mu_j\). The regularized effective dimension \[ d_{\mathrm{eff}}(\lambda) =\sum_{j\geq1}\frac{\mu_j}{\mu_j+\lambda} \] replaces the raw dimension in variance bounds. The estimation term is typically of order \(\sqrt{d_{\mathrm{eff}}(\lambda)/n}\), while regularization bias grows with \(\lambda\).
Choosing complexity
For a sieve, choose dimension so approximation and stochastic errors balance. For a penalty, choose \(\lambda_n\) large enough to dominate the localized empirical process but small enough that regularization bias vanishes. Data-dependent selection by cross-validation, Lepski's method, information criteria, or aggregation requires another uniform argument over the candidate tuning parameters; it is not automatically justified by a rate proved for one deterministic choice.