Posterior predictive checks ask whether data replicated from the fitted model resemble the observed data in scientifically relevant ways. They are model criticism, not proof that the model is true.
Replicated data
A posterior predictive check compares the observed data with data generated after conditioning on the observed data.
Definition. Let \(Y^{\mathrm{rep}}\) be drawn from \[ p(y^{\mathrm{rep}}\mid y)=\int p_\theta(y^{\mathrm{rep}})\Pi(\mathrm{d}\theta\mid y). \] For a discrepancy \(T\), the posterior predictive tail probability is \[ p_{\mathrm{ppc}}=\mathbb P\{T(Y^{\mathrm{rep}},\theta)\ge T(Y,\theta)\mid Y\}. \]
Remark. The discrepancy may depend on \(\theta\), but then both replicated and observed discrepancies must be computed under the same posterior draw of \(\theta\).
Double use of data
Posterior predictive \(p\)-values are usually conservative because the data are used both to fit and to check.
Proposition. Posterior predictive \(p\)-values are not generally uniform under the model.
Proof. Uniformity would require the reference distribution of the statistic to be fixed before seeing the data. Here the reference distribution is itself conditioned on the observed data, which pulls replications toward the observed features and reduces tail variation.
Graphical checks and residual structure
The most useful checks are often graphical: residual patterns, tail behavior, group discrepancies, and dependence not captured by the likelihood.
Example. In a regression model, one may compare residual maxima, residual autocorrelation, or groupwise variance between \(Y\) and \(Y^{\mathrm{rep}}\). A systematic mismatch points to a missing component of the likelihood, not merely to a bad parameter estimate.
Checks are conditional on the question
A model can pass one check and fail another. That is a feature, because different checks target different scientific failures.
Remark. Posterior predictive checking should be read alongside prior predictive checking: the first asks whether the fitted model can reproduce the observed data; the second asks whether the model was plausible before seeing the data.
Posterior predictive \(p\)-values
Definition. For discrepancy \(T(y,\theta)\), the posterior predictive \(p\)-value is \[ p_{\mathrm{ppc}} =P\{T(Y^{\mathrm{rep}},\theta) \ge T(y,\theta)\mid y\}, \] where \[ \theta\sim\Pi(\cdot\mid y),\qquad Y^{\mathrm{rep}}\sim P_\theta. \]
Proposition. Posterior predictive \(p\)-values are generally not uniform under the model and tend to be conservative because the observed data are used both to construct the posterior and to evaluate the discrepancy.
Explanation. Conditional on a fixed \(\theta\), the tail probability comparing iid \(Y^{\mathrm{rep}}\) and \(Y\) can be uniform for a continuous pivotal statistic. Posterior checking replaces the fixed \(\theta\) by a distribution learned from \(Y\), coupling the fitted reference distribution to the observed discrepancy. This dependence shrinks tail probabilities toward the center. Uniform calibration is therefore not the defining goal; detecting scientifically meaningful mismatch is.
Ancillary and conditional checks
Proposition. If a statistic \(A(Y)\) is ancillary for \(\theta\), so its law does not depend on \(\theta\), then its posterior predictive distribution equals its sampling distribution: \[ P\{A(Y^{\mathrm{rep}})\in B\mid y\} =P_\theta\{A(Y)\in B\}. \]
Proof. The integrand \(P_\theta\{A(Y)\in B\}\) is constant in \(\theta\). Integrating it against the posterior leaves the same constant.
Conditioning on sufficient statistics or using ancillary discrepancies can reduce double use of data, but exact ancillary structure is rare in complex hierarchical models. Leave-one-out predictive checks provide another route: observation \(i\) is compared with a predictive law fitted without \(y_i\).
Graphical checks by model component
A useful discrepancy isolates one model claim:
- tail counts, maxima, and robust scale for observation tails;
- group means and between-group variance for hierarchical pooling;
- autocorrelation and run lengths for time dependence;
- zero counts and upper quantiles for count dispersion;
- residual-versus-fitted smoothers for mean form;
- calibration curves for probabilistic prediction.
Plot replicated distributions, not only a scalar \(p\)-value. The direction and magnitude of mismatch often suggest the next model, whereas a thresholded number does not.
Simulation-based calibration
Theorem (rank uniformity). Draw \[ \theta^{(s)}\sim\pi,\qquad y^{(s)}\sim p_{\theta^{(s)}}, \] and then obtain \(L\) exact iid posterior draws \(\theta_1^{(s)},\ldots,\theta_L^{(s)}\). For a continuous scalar functional \(g\), the rank \[ R^{(s)} =\sum_{\ell=1}^L 1\{g(\theta_\ell^{(s)})<g(\theta^{(s)})\} \] is uniform on \(\{0,\ldots,L\}\).
Proof. Under the joint prior predictive law, \[ \theta^{(s)}\mid y^{(s)} \sim\Pi(\cdot\mid y^{(s)}). \] Conditional on \(y^{(s)}\), the generating parameter and the \(L\) posterior draws are therefore exchangeable. With no ties, every one of their \(L+1\) order positions has probability \(1/(L+1)\).
Simulation-based calibration checks an inference algorithm against a model: U-shaped rank histograms suggest underdispersed computation, hump-shaped histograms suggest overdispersion, and slopes suggest bias. It does not check whether the model describes real data.
Leave-one-out probability integral transforms
Definition. For continuous observations, define \[ u_i=F_{-i}(y_i), \qquad F_{-i}(y)=P(Y_i^{\mathrm{rep}}\le y\mid y_{-i}). \] Under conditional exchangeability and a calibrated model, the \(u_i\)'s are marginally uniform, though generally dependent.
Histograms reveal global miscalibration; plotting transformed residuals against covariates or time localizes structure. For discrete outcomes, randomized or interval probability transforms avoid artificial spikes.
Model expansion after failure
A failed check should lead to a controlled expansion: heavier tails, nonlinear predictors, group-specific variation, dependence, or an explicit observation process. Recheck both the targeted discrepancy and previously adequate features. An ever more flexible model can reproduce the sample while degrading prediction, so expansion should be evaluated by held-out scoring and scientific interpretability as well as in-sample replication.