Approximate inference replaces exact posterior integration by a tractable approximation. The approximation should be understood as a mathematical object with its own error, not as a different posterior.
Laplace approximation
Laplace approximation is local Gaussian approximation at a posterior mode.
Theorem. Let \(h\) have a strict interior maximum at \(\hat\theta\), and let \(H=-\nabla^2 h(\hat\theta)\) be positive definite. Then, formally, \[ \int \exp\{n h(\theta)\}a(\theta)\,\mathrm{d}\theta \approx \exp\{n h(\hat\theta)\}a(\hat\theta)(2\pi/n)^{d/2}\det(H)^{-1/2}. \]
Proof. Taylor expand \(h(\theta)\) around \(\hat\theta\): \[ h(\theta)=h(\hat\theta)-\frac12(\theta-\hat\theta)^\top H(\theta-\hat\theta)+o(\|\theta-\hat\theta\|^2). \] The change of variables \(u=\sqrt n(\theta-\hat\theta)\) reduces the leading term to a Gaussian integral.
Gaussian posterior approximation
In Bayesian computation, Laplace approximation turns curvature into covariance.
Proposition. If \(\hat\theta\) is the posterior mode and \(H_n=-\nabla^2\log\{\pi(\theta)p_\theta(X)\}|_{\hat\theta}\), then the Laplace approximation is \(N(\hat\theta,H_n^{-1})\).
Proof. Apply the quadratic Taylor expansion to the log posterior density and normalize the resulting Gaussian kernel.
Variational approximation
Variational methods choose an approximating distribution from a restricted family.
Definition. For a variational family \(\mathcal Q\), reverse-KL variational Bayes solves \[ q^*=\arg\min_{q\in\mathcal Q}\mathrm{KL}(q\,\|\,\Pi(\cdot\mid X)). \]
Remark. Laplace approximation is local and curvature-based; variational approximation is global over a restricted class but biased toward the geometry induced by the chosen divergence.
Approximation error
Every approximate posterior should be judged by the posterior functionals it is meant to approximate.
Proposition. If \(d_{\mathrm{TV}}(Q,\Pi(\cdot\mid X))\le\varepsilon\) and \(|g|\le M\), then \[ \left|\int g\,\mathrm{d}Q-\int g\,\mathrm{d}\Pi(\cdot\mid X)\right|\le 2M\varepsilon. \]
Proof. The total variation distance is half the \(L^1\) distance between densities under a common dominating measure. The expectation difference is bounded by \(M\) times that \(L^1\) distance.
Local Gaussian expansion
Theorem (Laplace posterior). Let \[ \ell_n^\pi(\theta)=\log p_\theta(x^{(n)})+\log\pi(\theta) \] have an interior maximizer \(\widehat\theta_{\mathrm{MAP}}\), and let \[ H_n=-\nabla^2\ell_n^\pi(\widehat\theta_{\mathrm{MAP}}) \] be positive definite. If third derivatives are controlled on the posterior concentration neighborhood, then the Laplace approximation is \[ \Pi(\mathrm{d}\theta\mid x) \approx N_d(\widehat\theta_{\mathrm{MAP}},H_n^{-1})(\mathrm{d}\theta). \]
Derivation. Taylor's theorem gives \[ \ell_n^\pi(\theta) =\ell_n^\pi(\widehat\theta_{\mathrm{MAP}}) -\frac12(\theta-\widehat\theta_{\mathrm{MAP}})^\top H_n(\theta-\widehat\theta_{\mathrm{MAP}}) +R_n(\theta). \] Neglecting \(R_n\) and normalizing the exponential quadratic gives the stated Gaussian. A rigorous approximation requires posterior mass outside the local region to vanish and the scaled remainder to be uniformly \(o(1)\).
Posterior mean correction
The mode and mean differ at the next order. In one dimension, write \[ \ell_n^\pi(\theta) =\ell_n^\pi(\widehat\theta) -\frac{H}{2}u^2+\frac{T}{6}u^3+\cdots, \qquad u=\theta-\widehat\theta. \] Expanding the exponential around the Gaussian kernel gives \[ E(\theta\mid x)-\widehat\theta \approx \frac{T}{2H^2}. \] Thus skewness, represented by the third derivative \(T\), shifts the posterior mean away from the mode. The shift is typically \(O(n^{-1})\) in regular fixed-dimensional models but can matter in small samples or near boundaries.
Laplace approximation for an integral
Theorem. Let \(f\) have a unique interior minimum \(x_0\) with \(f''(x_0)>0\), and let \(g\) be continuous with \(g(x_0)\ne0\). Then \[ \int e^{-nf(x)}g(x)\,\mathrm{d}x =e^{-nf(x_0)} g(x_0) \sqrt{\frac{2\pi}{nf''(x_0)}} \{1+o(1)\}. \]
Proof. Outside a small neighborhood of \(x_0\), uniqueness gives an exponentially smaller contribution. Inside it, \[ f(x)=f(x_0)+\frac12f''(x_0)(x-x_0)^2+o((x-x_0)^2). \] Set \(z=\sqrt{nf''(x_0)}(x-x_0)\), use dominated convergence on expanding bounded regions, and control the Gaussian tails.
Boundary and multimodal failures
Example. If a variance component \(\tau\ge0\) has posterior mode at \(\tau=0\), a symmetric normal approximation puts half its mass on impossible negative values. Transforming to \(\log\tau\) does not repair a genuine boundary mode at zero. One must use a boundary-aware approximation, mixture representation, numerical integration, or simulation.
For separated modes \(\widehat\theta_k\), a multimodal Laplace approximation sums local contributions: \[ m(x)\approx \sum_k e^{\ell_n^\pi(\widehat\theta_k)} (2\pi)^{d/2}|H_k|^{-1/2}. \] A single-mode Gaussian can badly distort posterior probabilities even when each local expansion is accurate.
Approximation metrics
Proposition (Pinsker consequence). If \[ \operatorname{KL}(Q\|\Pi)\le\varepsilon, \] then \[ \|Q-\Pi\|_{\mathrm{TV}} \le\sqrt{\varepsilon/2}, \] and for \(|g|\le M\), \[ |E_Qg-E_\Pi g| \le M\sqrt{2\varepsilon}. \]
Proof. Pinsker's inequality gives the total-variation bound. The dual characterization \[ |E_Qg-E_\Pi g| \le2M\|Q-\Pi\|_{\mathrm{TV}} \] gives the second inequality.
A small reverse KL is therefore strong enough to control bounded expectations, but the optimized ELBO does not reveal the unknown KL value because it differs from \(\log m(x)\) by that value. Internal convergence of an optimization algorithm is not a certificate of posterior accuracy.
Laplace versus variational approximation
Laplace uses local curvature at a mode and is asymptotically accurate in regular unimodal fixed-dimensional models. Variational inference chooses the closest member of a restricted family under a specified divergence and can represent non-Gaussian shapes if the family is rich. Laplace has no global optimization gap after a mode is found but can miss other modes; variational methods can cover larger models but add family restriction and optimization error. Both should be checked against simulation, importance correction, or known marginals on representative datasets.