Monte Carlo methods approximate posterior expectations by random averages. The target is mathematical integration; simulation error must therefore be quantified like any other approximation error.
Posterior expectations as integrals
Most Bayesian summaries are expectations under the posterior.
Definition. For a posterior \(\Pi(\mathrm{d}\theta\mid X)\) and integrable function \(g\), the target is \[ I=\int g(\theta)\Pi(\mathrm{d}\theta\mid X). \] Given draws \(\theta_1,\ldots,\theta_N\), the Monte Carlo estimator is \(\hat I_N=N^{-1}\sum_{s=1}^N g(\theta_s)\).
Theorem. If \(\theta_s\) are iid from the posterior and \(\mathbb E|g(\theta)|<\infty\), then \(\hat I_N\to I\) almost surely.
Proof. This is the strong law of large numbers applied to the iid sequence \(g(\theta_s)\).
Monte Carlo central limit theorem
Error bars require second moments, not just consistency.
Proposition. If \(\operatorname{Var}\{g(\theta)\}<\infty\), then \[ \sqrt N(\hat I_N-I)\Rightarrow N(0,\operatorname{Var}\{g(\theta)\}). \]
Proof. Apply the classical central limit theorem to \(g(\theta_s)\). A consistent estimate of the variance gives standard Monte Carlo error estimates.
Importance sampling
Importance sampling changes the simulation law but corrects the integral with weights.
Definition. If \(q\) dominates the posterior density \(\pi\), then \[ I=\frac{\int g(\theta)w(\theta)q(\theta)\,\mathrm{d}\theta}{\int w(\theta)q(\theta)\,\mathrm{d}\theta}, \qquad w(\theta)=\frac{\pi(\theta\mid X)}{q(\theta)}. \]
Remark. Importance sampling fails when the weight distribution has heavy tails. A proposal must cover the posterior tails at least as well as the target.
Effective sample size
Effective sample size is a diagnostic for weight or dependence degeneracy, not a new theorem about the posterior.
Proposition. For normalized importance weights \(\bar w_s=w_s/\sum_r w_r\), the common diagnostic \[ N_{\mathrm{eff}}=\frac{1}{\sum_s\bar w_s^2} \] equals \(N\) for equal weights and approaches \(1\) when one draw dominates.
Proof. The expression is the reciprocal of the squared \(\ell^2\)-norm of the normalized weight vector. It is maximized by the uniform vector and minimized by a point mass.
Independent Monte Carlo error
Theorem. Let \(\theta_1,\ldots,\theta_N\stackrel{\mathrm{iid}}{\sim}\Pi(\cdot\mid x)\) and \[ \mu=E\{g(\theta)\mid x\},\qquad \sigma_g^2=\operatorname{Var}\{g(\theta)\mid x\}<\infty. \] Then \[ \widehat\mu_N=\frac1N\sum_{s=1}^Ng(\theta_s) \] is unbiased, \[ \operatorname{Var}(\widehat\mu_N\mid x) =\frac{\sigma_g^2}{N}, \] and \[ \sqrt N(\widehat\mu_N-\mu) \Rightarrow N(0,\sigma_g^2). \]
Proof. Linearity gives unbiasedness. Independence eliminates cross-covariances, giving the variance. The iid central limit theorem gives the Gaussian limit.
Corollary. With sample variance \[ s_g^2=\frac1{N-1}\sum_s\{g(\theta_s)-\widehat\mu_N\}^2, \] a Monte Carlo standard error is \[ \operatorname{MCSE}(\widehat\mu_N)=s_g/\sqrt N. \] This measures numerical integration uncertainty conditional on the observed data; it is distinct from posterior standard deviation \(\sigma_g\).
Estimating tail probabilities and quantiles
Proposition. For posterior event \(A\), the Monte Carlo estimator \[ \widehat p_N=\frac1N\sum_s1\{\theta_s\in A\} \] has standard error approximately \[ \sqrt{\frac{p(1-p)}{N}}. \] To estimate a small probability \(p\) with relative standard error at most \(r\), one needs approximately \[ N\ge\frac{1-p}{pr^2} \] independent draws.
Proof. The indicators are Bernoulli\((p)\), so \(\operatorname{Var}(\widehat p_N)=p(1-p)/N\). Divide its standard deviation by \(p\) and solve the inequality.
Tail accuracy can therefore require far more draws than posterior means. A statement about a \(0.001\) posterior tail based on a few thousand draws is numerically fragile even if the chain otherwise appears stable.
Importance sampling identities
Theorem. Let the posterior density be \[ \pi(\theta\mid x)=\frac{\widetilde\pi(\theta)}{Z}, \qquad Z=\int\widetilde\pi(\theta)\,\mathrm{d}\theta, \] and let proposal \(q\) dominate \(\widetilde\pi\). If \(\theta_s\stackrel{\mathrm{iid}}{\sim}q\) and \[ w(\theta)=\frac{\widetilde\pi(\theta)}{q(\theta)}, \] then the self-normalized estimator \[ \widehat\mu_{\mathrm{SN}} =\frac{\sum_sw_sg(\theta_s)}{\sum_sw_s} \] converges almost surely to \(E_\pi g\), provided \(E_q|wg|<\infty\) and \(E_qw<\infty\).
Proof. The strong law gives \[ \frac1N\sum_sw_sg(\theta_s) \to E_q(wg)=Z E_\pi g, \] \[ \frac1N\sum_sw_s\to E_qw=Z. \] Their ratio converges to \(E_\pi g\).
Theorem (self-normalized IS CLT). If \[ E_q\left[ w(\theta)^2\{g(\theta)-\mu\}^2 \right]<\infty, \] then \[ \sqrt N(\widehat\mu_{\mathrm{SN}}-\mu) \Rightarrow N\left( 0,\ \frac{ E_q[w^2(g-\mu)^2]} {(E_qw)^2} \right). \]
Proof. Apply the bivariate CLT to sample means of \((wg,w)\), then the delta method to the ratio map \(a/b\). The gradient at \((ZE_\pi g,Z)\) is \((1/Z,-\mu/Z)\), producing the displayed variance.
Weight-tail diagnostics
The heuristic \[ N_{\mathrm{eff}} =\frac{(\sum_sw_s)^2}{\sum_sw_s^2} \] summarizes weight concentration but is not a universal variance formula. Two integrands can have very different errors under the same weights. More fundamentally, if \[ E_qw^2=\infty, \] the ordinary importance-sampling CLT can fail. A proposal must have tails at least as heavy as the target in every important direction.
Example. Target \(N(0,1)\) with proposal \(N(0,\sigma^2)\). The second weight moment is finite only when \[ \sigma^2>\frac12. \] Indeed, \[ E_qw^2 \propto\int \exp\left\{-\theta^2+ \frac{\theta^2}{2\sigma^2}\right\} \mathrm{d}\theta, \] which is integrable exactly when \(1-1/(2\sigma^2)>0\). An overly narrow proposal can therefore have infinite variance despite covering the target mode.
Bridge sampling for normalizing constants
Proposition (bridge identity). Let \(\widetilde\pi_1,\widetilde\pi_0\) have normalizing constants \(Z_1,Z_0\), and let \(h>0\). Then \[ \frac{Z_1}{Z_0} =\frac{ E_{\pi_0}\{h(\theta)\widetilde\pi_1(\theta)\} }{ E_{\pi_1}\{h(\theta)\widetilde\pi_0(\theta)\} }. \]
Proof. Both numerator and denominator integrate the common quantity \(h(\theta)\widetilde\pi_0(\theta)\widetilde\pi_1(\theta)\); the first is divided by \(Z_0\) and the second by \(Z_1\). Taking their ratio gives \(Z_1/Z_0\).
Bridge sampling can estimate marginal likelihoods when posterior and reference samples overlap well. Harmonic-mean estimators correspond to a poor bridge and often have infinite variance; they should not be used merely because they require no additional simulation.