Maximum likelihood turns the sample log-likelihood into a random optimization problem. Its familiar Gaussian limit is not a consequence of maximization alone: it combines identification of the population criterion, uniform stochastic convergence, local differentiability, a score central limit theorem, and nonsingular information.
Exact and approximate maximizers
Definition. An exact maximum-likelihood estimator satisfies \[ \ell_n(\hat\theta_n)=\sup_{\theta\in\Theta}\ell_n(\theta). \] For consistency it is enough that \[ \ell_n(\hat\theta_n)\ge\sup_{\theta\in\Theta}\ell_n(\theta)-o_p(n), \] while root-\(n\) distribution theory commonly requires an optimization error \(o_p(1)\) on the log-likelihood scale.
An exact maximizer need not exist on a noncompact or open parameter space. A measurable approximate maximizer is therefore the natural asymptotic object. In mixture models the likelihood may even be unbounded, so constraints or penalties are part of the model specification rather than mere numerical conveniences.
Invariance of the MLE. If \(\hat\theta\) maximizes \(L(\theta;x)\) and \(\psi=g(\theta)\), then the profile MLE \[ \hat\psi\in\arg\max_{\psi}\sup_{\theta:g(\theta)=\psi}L(\theta;x) \] contains \(g(\hat\theta)\). If \(g\) is one-to-one, \(\hat\psi=g(\hat\theta)\).
Proof. The fiber containing \(\hat\theta\) has profile likelihood at least \(L(\hat\theta;x)\), while no fiber can have a larger value than the global supremum. Hence \(g(\hat\theta)\) is a profile maximizer.
Consistency by separation of the population criterion
Argmax consistency theorem. Let \(M_n:\Theta\to\mathbb R\) be random criteria and \(M:\Theta\to\mathbb R\). Suppose, for every \(\varepsilon>0\), \[ \sup_{\theta\in\Theta}|M_n(\theta)-M(\theta)|\to_p0, \] \[ M(\theta_0)> \sup_{\{\theta:d(\theta,\theta_0)\ge\varepsilon\}}M(\theta), \] and \[ M_n(\hat\theta_n)\ge\sup_{\theta\in\Theta}M_n(\theta)-o_p(1). \] Then \(\hat\theta_n\to_p\theta_0\).
Proof. Fix \(\varepsilon>0\) and let \[ 3\delta=M(\theta_0)- \sup_{d(\theta,\theta_0)\ge\varepsilon}M(\theta)>0. \] On the event \(\sup_\theta|M_n-M|\lt\delta\) and with optimization error below \(\delta\), \[ M_n(\hat\theta_n)\ge M_n(\theta_0)-\delta >M(\theta_0)-2\delta, \] whereas every \(\theta\) outside the \(\varepsilon\)-ball satisfies \[ M_n(\theta)\le M(\theta)+\delta \le M(\theta_0)-2\delta. \] Hence \(\hat\theta_n\) lies inside the ball. The event has probability tending to one.
For i.i.d. likelihood, take \(M_n(\theta)=P_n\log p_\theta\) and \(M(\theta)=P_0\log p_\theta\). Under correct specification, the Kullback-Leibler identity makes \(\theta_0\) the unique maximizer when the model is identifiable. A uniform law of large numbers supplies the first condition; compactness plus continuity is a convenient sufficient route, not an indispensable assumption.
Example (Bernoulli likelihood). If \(S_n=\sum_iX_i\), then \[ \ell_n(p)=S_n\log p+(n-S_n)\log(1-p). \] For \(0\lt S_n\lt n\), the score vanishes at \(\hat p=S_n/n\), and \[ \ell_n''(p)=-\frac{S_n}{p^2}-\frac{n-S_n}{(1-p)^2}\lt0. \] At \(S_n=0\) or \(n\), the maximizer lies on the boundary. The law of large numbers still gives \(\hat p\to p_0\).
Root-\(n\) expansion
Asymptotic normality of a regular MLE. Let \(\theta_0\) be an interior point. Assume \(\hat\theta_n\to_p\theta_0\), \(S_n(\hat\theta_n)=o_p(\sqrt n)\), and, uniformly for \(\theta\) in a shrinking neighborhood of \(\theta_0\), \[ -\frac1n\nabla_\theta^2\ell_n(\theta)\to_p I(\theta_0), \] where \(I(\theta_0)\) is nonsingular. If \[ \frac1{\sqrt n}S_n(\theta_0)\rightsquigarrow N\{0,I(\theta_0)\}, \] then \[ \sqrt n(\hat\theta_n-\theta_0) =I(\theta_0)^{-1}\frac1{\sqrt n}S_n(\theta_0)+o_p(1) \rightsquigarrow N\{0,I(\theta_0)^{-1}\}. \]
Proof. Taylor's theorem in integral form gives \[ S_n(\hat\theta_n) =S_n(\theta_0) +\left\{\int_0^1\nabla^2\ell_n( \theta_0+t(\hat\theta_n-\theta_0))\,\mathrm{d}t\right\} (\hat\theta_n-\theta_0). \] Divide by \(\sqrt n\), move the Hessian term to the other side, and use consistency plus the uniform Hessian convergence. The inverse converges to \(I(\theta_0)^{-1}\); the approximate score equation and Slutsky's theorem yield the expansion and limit.
Corollary (estimated covariance). If \[ \hat I_n=-n^{-1}\nabla^2\ell_n(\hat\theta_n)\to_pI(\theta_0), \] then \[ \hat I_n^{1/2}\sqrt n(\hat\theta_n-\theta_0) \rightsquigarrow N(0,I_d). \] The inverse observed information \(J_n(\hat\theta_n)^{-1}\) estimates the covariance of \(\hat\theta_n\).
Local quadratic likelihood
For bounded \(h\), regularity gives the expansion \[ \ell_n(\theta_0+h/\sqrt n)-\ell_n(\theta_0) =h^\top\Delta_n-\frac12h^\top I(\theta_0)h+o_p(1), \qquad \Delta_n=\frac1{\sqrt n}S_n(\theta_0). \] Maximizing the quadratic approximation gives \(h=I^{-1}\Delta_n\), exactly the influence expansion above. This local likelihood geometry becomes the local asymptotic normality property on page 25.
Nuisance parameters and profile likelihood
Write \(\theta=(\psi^\top,\lambda^\top)^\top\). For fixed \(\psi\), let \[ \hat\lambda_\psi\in\arg\max_\lambda\ell_n(\psi,\lambda), \qquad \ell_n^{\mathrm{prof}}(\psi)=\ell_n(\psi,\hat\lambda_\psi). \] The profile score has first-order information equal to the Schur complement \[ I_{\psi\cdot\lambda} =I_{\psi\psi} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. \] This is smaller than \(I_{\psi\psi}\): estimating nuisance parameters consumes information.
Profile-score identity. At points where \(\hat\lambda_\psi\) is differentiable and solves \(\partial_\lambda\ell_n(\psi,\hat\lambda_\psi)=0\), \[ \frac{\mathrm{d}}{\mathrm{d}\psi}\ell_n^{\mathrm{prof}}(\psi) =\partial_\psi\ell_n(\psi,\hat\lambda_\psi). \]
Proof. The chain rule gives \[ \frac{\mathrm{d}}{\mathrm{d}\psi}\ell_n(\psi,\hat\lambda_\psi) =\partial_\psi\ell_n+ \partial_\lambda\ell_n\,D\hat\lambda_\psi. \] The second term vanishes by the constrained score equation.
Boundary and singular likelihoods
If \(\theta_0\) lies on a boundary, the local parameter set is a cone rather than all of \(\mathbb R^d\); likelihood-ratio limits may be mixtures of chi-square laws. If information is singular, the quadratic approximation cannot identify all directions and rates can differ from \(\sqrt n\). In finite mixtures, change-points, support-boundary models, and weakly identified models, the regular MLE theorem must not be invoked by analogy. The local geometry of the particular experiment determines the rate and limit.