In statistics the objective is random. Optimization determines how well the sample criterion is solved; probability determines how that criterion differs from its population target. Keeping these two layers separate prevents an algorithmic convergence statement from being mistaken for statistical consistency.
Population and empirical criteria
Definition. For observations \(X_1,\ldots,X_n\) and losses \(\ell_\theta\), write \[ M(\theta)=P\ell_\theta=\mathbb E[\ell_\theta(X)], \qquad M_n(\theta)=P_n\ell_\theta =\frac1n\sum_{i=1}^n\ell_\theta(X_i). \] A population target satisfies \(\theta_0\in\operatorname*{arg\,min}_{\theta\in\Theta}M(\theta)\). An approximate \(M\)-estimator satisfies \[ M_n(\widehat\theta_n) \le\inf_{\theta\in\Theta}M_n(\theta)+\varepsilon_n. \]
The population minimizer can be meaningful under misspecification: it is the best approximation to the data-generating law under the chosen loss, not necessarily a parameter that generated the observations.
Uniform convergence and consistency
Argmin consistency. Suppose \(\Theta\) is compact, \(M\) has unique minimizer \(\theta_0\), and for every \(\delta>0\), \[ \inf_{\|\theta-\theta_0\|\ge\delta} [M(\theta)-M(\theta_0)]>0. \] If \[ \sup_{\theta\in\Theta}|M_n(\theta)-M(\theta)| \xrightarrow{P}0,\qquad \varepsilon_n=o_P(1), \] then \(\widehat\theta_n\xrightarrow{P}\theta_0\).
Proof. Fix \(\delta>0\) and let the displayed separation be \(\eta_\delta>0\). On the event \[ 2\sup_\Theta|M_n-M|+\varepsilon_n<\eta_\delta, \] every \(\theta\) outside \(B(\theta_0,\delta)\) satisfies \[ M_n(\theta) >M_n(\theta_0)+\varepsilon_n, \] and therefore cannot be an approximate minimizer. The probability of this event tends to one, so \(P(\|\widehat\theta_n-\theta_0\|\ge\delta)\to0\).
Pointwise laws of large numbers are insufficient when the minimizer is chosen after looking over all \(\theta\). Uniformity is what controls that data-dependent choice. Empirical-process methods supply such bounds for large classes of losses.
Local curvature converts criterion error to parameter error
Deterministic localization. Suppose on a neighborhood of \(\theta_0\), \[ M(\theta)-M(\theta_0) \ge c\|\theta-\theta_0\|^\alpha \] for \(c>0,\alpha\ge1\). If an estimator lies in that neighborhood and satisfies \(M(\widehat\theta)-M(\theta_0)\le\delta\), then \[ \|\widehat\theta-\theta_0\| \le(\delta/c)^{1/\alpha}. \]
Quadratic curvature corresponds to \(\alpha=2\). Sharp nonsmooth risks can have \(\alpha=1\). The curvature exponent changes how stochastic objective fluctuations translate into an estimation rate.
Score expansion and asymptotic linearity
Smooth \(M\)-estimator expansion. Suppose \(\widehat\theta_n\to_P\theta_0\), \(\nabla M(\theta_0)=0\), \(V=\nabla^2M(\theta_0)\) is nonsingular, and a uniform local Hessian law permits Taylor expansion of \(\nabla M_n\). If \(\nabla M_n(\widehat\theta_n)=o_P(n^{-1/2})\), then \[ \sqrt n(\widehat\theta_n-\theta_0) =-V^{-1}\sqrt n\,\nabla M_n(\theta_0)+o_P(1). \] If \(\sqrt n\,\nabla M_n(\theta_0)\Rightarrow N(0,\Sigma)\), the limit covariance is \(V^{-1}\Sigma V^{-\top}\).
Proof. The mean-value expansion is \[ \nabla M_n(\widehat\theta_n) =\nabla M_n(\theta_0) +V_n(\widetilde\theta_n) (\widehat\theta_n-\theta_0), \] where \(\widetilde\theta_n\) lies on the connecting segment. Consistency and the local Hessian law give \(V_n(\widetilde\theta_n)=V+o_P(1)\). Rearrange, multiply by \(\sqrt n\), and apply Slutsky's theorem.
The required empirical-process conditions and nonsmooth extensions are developed in M- and Z-estimation and empirical-process consistency. The optimization requirement \(\nabla M_n(\widehat\theta_n)=o_P(n^{-1/2})\) makes the computational tolerance explicit.