Compressed posterior summaries
Posterior compression replaces a complicated posterior object by a smaller representation whose fidelity can be checked. In redundant-predictor problems, regions or groups may be more stable than exact supports.
Definition. Let \(P\) be the full posterior predictive or posterior law and \(Q\) a compressed approximation. A density-ratio diagnostic studies \(r=\mathrm{d}P/\mathrm{d}Q\) on the sigma-field relevant to the summary. Exact fidelity is \(r=1\) \(Q\)-almost surely.
Proposition. If \(0<c\le r\le C<\infty\), then for any bounded measurable \(f\), \[ |E_Pf-E_Qf|\le \|f\|_\infty E_Q|r-1|. \]
Proof. Since \(E_Pf-E_Qf=E_Q[(r-1)f]\), the result follows by the triangle inequality and \(|f|\le\|f\|_\infty\).
Interpretability versus fidelity
Remark. A compressed summary is useful only when it preserves the inferential quantity at stake. Support compression, predictive compression, and credible-region compression are different mathematical targets.
Coresets and weighted likelihoods
Definition. A Bayesian coreset replaces \(\ell_n(\theta)=\sum_{i=1}^n\ell_i(\theta)\) by a weighted sparse sum \(\ell_w(\theta)=\sum_{i=1}^n w_i\ell_i(\theta)\), with many \(w_i=0\).
Proposition. If \(\sup_\theta|\ell_w(\theta)-\ell_n(\theta)|\le\varepsilon\), then the normalizing constants and posterior expectations of bounded functions are stable up to errors controlled by \(e^\varepsilon-1\).
Proof. The likelihood ratio between weighted and full posterior kernels lies between \(e^{-\varepsilon}\) and \(e^\varepsilon\). Normalize both measures and apply the same multiplicative bound to integrals of bounded test functions.
Posterior perturbation from log-kernel error
Theorem. Let \[ \pi(\theta)=\frac{e^{\ell(\theta)}p(\theta)}{Z}, \qquad \widetilde\pi(\theta) =\frac{e^{\widetilde\ell(\theta)}p(\theta)} {\widetilde Z}, \] and suppose \[ \sup_\theta|\widetilde\ell(\theta)-\ell(\theta)| \le\varepsilon. \] Then \[ e^{-2\varepsilon} \le\frac{\widetilde\pi(\theta)}{\pi(\theta)} \le e^{2\varepsilon}, \] \[ \|\widetilde\Pi-\Pi\|_{\mathrm{TV}} \le\frac12(e^{2\varepsilon}-1), \] and \[ \operatorname{KL}(\widetilde\Pi\|\Pi) \le2\varepsilon. \]
Proof. Kernel bounds imply \[ e^{-\varepsilon}Z \le\widetilde Z\le e^\varepsilon Z. \] Combining numerator and normalizer bounds gives the density-ratio bounds. Integrate \(|\widetilde\pi/\pi-1|\) for TV. For KL, the log density ratio is at most \(2\varepsilon\) pointwise.
Uniform log-likelihood approximation is strong and often unrealistic over an unbounded parameter space. Local posterior-weighted norms can yield sharper results if one separately proves that both posteriors remain in the controlled region.
Coreset construction in a Hilbert geometry
Definition. Center log-likelihood contributions \[ \widetilde\ell_i(\theta) =\ell_i(\theta)-E_\nu\ell_i(\theta) \] in a reference Hilbert space \(L^2(\nu)\). A coreset chooses nonnegative sparse weights \(w\) to minimize \[ \left\| \sum_{i=1}^n\widetilde\ell_i -\sum_{i=1}^nw_i\widetilde\ell_i \right\|_{L^2(\nu)}. \]
Proposition. If the residual log-likelihood function has \(L^2(\nu)\)-norm \(\delta\), then its mean-squared error under \(\nu\) is \(\delta^2\), but no uniform posterior bound follows without relating \(\nu\) to posterior mass and controlling residual tails.
Greedy Hilbert-coreset methods select contributions aligned with the current residual; sparse nonnegative least squares then refines weights. The reference distribution should cover posterior-relevant regions. A poor pilot can certify approximation in the wrong geometry.
Local asymptotic preservation
Theorem (LAN preservation). Suppose the full and compressed log likelihoods satisfy, uniformly for bounded \(h\), \[ \ell_w(\theta_0+h/\sqrt n) -\ell_w(\theta_0) \] \[ = \ell_n(\theta_0+h/\sqrt n) -\ell_n(\theta_0) +o_P(1). \] If the full posterior satisfies regular BvM and the compressed posterior also localizes at root-\(n\) rate, then both have the same first-order Gaussian limit.
Proof. The uniform difference makes the compressed likelihood share the same LAN score and information. Prior continuity is unchanged. Apply the BvM normalization proof to the compressed posterior; localization controls the region outside the shared expansion.
Matching the total log likelihood up to \(o(n)\) is insufficient for BvM; the local difference must be \(o(1)\) on root-\(n\) neighborhoods.
Density-ratio estimation by classification
Theorem. Let samples come from \(P\) with class label \(Y=1\) and from \(Q\) with \(Y=0\), using equal class probabilities. The Bayes classifier satisfies \[ \eta(x)=P(Y=1\mid X=x) =\frac{p(x)}{p(x)+q(x)}, \] so \[ \frac{p(x)}{q(x)} =\frac{\eta(x)}{1-\eta(x)}. \]
Proof. Apply Bayes' formula with equal class priors and rearrange.
Training a calibrated probabilistic classifier to distinguish exact-posterior reference draws from approximate draws estimates a density ratio. Near-perfect classification indicates strong discrepancy; chance-level classification is only evidence relative to the classifier class and sample size.
Importance correction of an approximate posterior
Definition. If \(\theta_s\sim Q\) and the exact unnormalized posterior is \(\widetilde\pi\), use \[ w_s=\frac{\widetilde\pi(\theta_s)}{q(\theta_s)} \] to estimate exact posterior expectations by normalized weighting.
Proposition. The chi-square divergence controls asymptotic weight efficiency: \[ E_Q\left( \frac{\pi(\theta)}{q(\theta)} \right)^2 =1+\chi^2(\Pi\|Q). \] Thus an idealized effective-sample fraction is approximately \[ \frac1{1+\chi^2(\Pi\|Q)}. \]
Proof. By definition, \[ \chi^2(\Pi\|Q) =\int\left(\frac{\pi}{q}-1\right)^2q =\int\frac{\pi^2}{q}-1, \] because \(\int\pi=\int q=1\).
Reverse-KL variational approximations can omit target tails, producing infinite \(\chi^2(\Pi\|Q)\) even when ELBO is good. Importance correction then fails rather than automatically repairing the approximation.
Compression targets
Different compressions preserve different objects:
- coresets approximate the likelihood or posterior;
- posterior distillation approximates draws or predictive distributions;
- projection predictive methods compress predictor sets;
- low-rank Gaussian approximations compress covariance;
- summary networks compress data for simulation-based inference.
A guarantee should name the retained metric: local likelihood ratio, KL, Wasserstein, bounded decisions, predictive score, or selected functional. “Posterior compression” alone is not a mathematical target.
Diagnostics at three levels
- Kernel level: compare log likelihoods, scores, and Hessians on posterior-relevant draws.
- Distribution level: use density-ratio classification, KSD, moments, tails, and importance-weight diagnostics.
- Decision level: compare predictions, credible limits, and posterior expected losses for the actual task.
Agreement at one level need not imply agreement at another. Local score matching can preserve BvM while missing remote modes; predictive agreement can coexist with different parameter posteriors under nonidentifiability.