Under misspecification, the posterior often concentrates around the KL projection of the truth rather than a true parameter. The limiting covariance then differs from ordinary Fisher information.
Pseudo-true parameters
When the model is wrong, the target becomes the best approximation inside the model.
Definition. Let \(P_0\) be the data-generating law and \(P_\theta\) the model. A pseudo-true parameter is \[ \theta^*=\arg\min_{\theta}K(P_0,P_\theta) =\arg\max_\theta P_0\log p_\theta, \] when the minimizer exists.
Proposition. The expected score at an interior pseudo-true parameter is zero: \(P_0\dot\ell_{\theta^*}=0\).
Proof. Differentiate the criterion \(\theta\mapsto P_0\log p_\theta\) at its interior maximum and use the regularity that permits interchange of differentiation and integration.
Sandwich covariance
The curvature and score variance no longer agree under misspecification.
Definition. Let \[ J=-P_0\ddot\ell_{\theta^*},\qquad V=P_0(\dot\ell_{\theta^*}\dot\ell_{\theta^*}^\top). \] The frequentist covariance of the maximum likelihood estimator is \(J^{-1}VJ^{-1}\), while the local posterior covariance is often \(J^{-1}\).
Remark. This mismatch is why credible sets can have incorrect frequentist coverage under misspecification even when the posterior is asymptotically normal.
Misspecified BvM
The posterior can still be asymptotically Gaussian, but it is centered at the pseudo-true parameter.
Theorem. Under smoothness, prior positivity at \(\theta^*\), posterior localization, and nonsingular \(J\), the posterior of \(\sqrt n(\theta-\hat\theta_n)\) is asymptotically \(N(0,J^{-1})\).
Proof. Expand the empirical log likelihood around \(\hat\theta_n\). The first derivative vanishes at the empirical maximizer, and the Hessian converges to \(-J\). Prior continuity makes the prior locally constant. Laplace approximation then gives the Gaussian posterior covariance \(J^{-1}\).
Robust adjustment
If frequentist calibration is desired under misspecification, posterior covariance may need sandwich correction.
Remark. Bayesian coherence and frequentist calibration are different targets. A sandwich-adjusted posterior is no longer the exact posterior but can be useful as a calibrated uncertainty approximation.
Kullback-Leibler projection
Definition. If the true density \(p_0\) need not belong to \(\{p_\theta:\theta\in\Theta\}\), a pseudo-true parameter is \[ \theta^\star \in\arg\min_{\theta\in\Theta} K(p_0,p_\theta) =\arg\max_{\theta\in\Theta} P_0\log p_\theta. \]
Proposition. If \(\theta^\star\) is an interior differentiability point, then \[ P_0\dot\ell_{\theta^\star}=0. \]
Proof. Differentiate the population objective \(P_0\log p_\theta\) at its interior maximizer, interchanging derivative and expectation under the regularity assumption.
Two information matrices
Definition. At \(\theta^\star\), define \[ I=-P_0\ddot\ell_{\theta^\star}, \qquad J=P_0\dot\ell_{\theta^\star} \dot\ell_{\theta^\star}^\top. \] Under correct specification, the information identity gives \(I=J\). Under misspecification, they generally differ.
Theorem (M-estimator sandwich limit). If \(\widehat\theta\) maximizes empirical log likelihood and standard M-estimation conditions hold, then \[ \sqrt n(\widehat\theta-\theta^\star) \Rightarrow N(0,I^{-1}JI^{-\top}). \]
Proof. Taylor-expand the empirical score: \[ 0=\mathbb P_n\dot\ell_{\widehat\theta} =\mathbb P_n\dot\ell_{\theta^\star} +\mathbb P_n\ddot\ell_{\widetilde\theta} (\widehat\theta-\theta^\star). \] The Hessian converges to \(-I\), while the CLT gives \[ \sqrt n\mathbb P_n\dot\ell_{\theta^\star} \Rightarrow N(0,J). \] Solve and apply Slutsky's theorem.
Misspecified local asymptotic normality
Theorem (misspecified BvM). Under posterior concentration at \(\theta^\star\), local quadratic likelihood expansion, and a prior positive and continuous at \(\theta^\star\), \[ \Pi\left( \sqrt n(\theta-\widehat\theta) \in\cdot\mid X^{(n)} \right) \approx N(0,I^{-1}) \] in total variation. The exact posterior covariance is therefore \(I^{-1}/n\), while repeated-sampling variation of its center is \(I^{-1}JI^{-\top}/n\).
Proof outline. The empirical log likelihood around \(\theta^\star+h/\sqrt n\) expands as \[ h^\top\Delta_n-\frac12h^\top Ih+o_P(1), \qquad \Delta_n=n^{-1/2}\sum_i\dot\ell_{\theta^\star}(X_i) \Rightarrow N(0,J). \] Conditional on the data, completing the square gives covariance \(I^{-1}\) and random center \(I^{-1}\Delta_n\). Localization and prior continuity complete the same normalization argument as regular BvM.
Coverage mismatch
Proposition. A nominal posterior ellipsoid based on \(I^{-1}\) generally lacks nominal frequentist coverage for \(\theta^\star\) unless \(I=J\) or the target direction happens to satisfy the corresponding covariance equality.
Proof. Posterior quantiles use the quadratic form induced by \(I\), whereas \(\sqrt n(\widehat\theta-\theta^\star)\) has sandwich covariance. The resulting quadratic form is not \(\chi_d^2\) unless the covariance matches \(I^{-1}\).
Open-faced sandwich transformation
Definition. Suppose posterior draws have local covariance \(I^{-1}/n\). Let \(C\) satisfy \[ C I^{-1}C^\top =I^{-1}JI^{-\top}. \] Transform centered draws by \[ \theta^{(s)}_{\mathrm{adj}} =\widehat\theta +C\{\theta^{(s)}-\widehat\theta\}. \]
Proposition. The adjusted draws have asymptotic covariance \[ \frac1nI^{-1}JI^{-\top}. \]
Proof. Covariance transforms by \[ C(I^{-1}/n)C^\top. \] Use the defining equation for \(C\).
One possible matrix choice uses symmetric square roots: \[ C=(I^{-1}JI^{-\top})^{1/2} (I^{-1})^{-1/2}. \] The adjustment targets frequentist covariance around the pseudo-true parameter; it is not an exact posterior under the original model and need not repair biased prediction.
Generalized Bayes and learning rates
Definition. A tempered or generalized posterior is \[ \Pi_\eta(\mathrm{d}\theta\mid X^{(n)}) \propto \exp\{-\eta n\mathbb P_n\ell_\theta\} \Pi(\mathrm{d}\theta), \qquad \eta>0. \] Changing \(\eta\) rescales local posterior covariance but does not generally reproduce a full sandwich matrix when \(d>1\).
A scalar temperature can calibrate one average direction or predictive criterion, while matrix-valued misspecification requires more structure. Robust likelihoods, explicit discrepancy models, nonparametric model expansion, and targeted estimating-equation posteriors address different failure modes. The inferential target should be named before choosing a correction.
Misspecified prediction
The pseudo-true parameter minimizes expected log loss within the model, so \[ p_{\theta^\star} \] is the best model member under KL divergence. Posterior predictive uncertainty around it shrinks with \(n\), but it does not include structural discrepancy \[ p_0-p_{\theta^\star}. \] Posterior predictive checks and out-of-sample scoring remain necessary even when misspecified BvM holds perfectly.