Prediction and marginal likelihood are obtained by integrating out the unknown parameter. The same normalizing constant that makes the posterior a probability measure also scores the model's predictive fit.
Prior predictive and posterior predictive laws
Before observing data, uncertainty is averaged with the prior. After observing data, the averaging measure becomes the posterior.
Definition. For a future observation \(Y\) with conditional law \(Q_\theta\), the prior predictive and posterior predictive laws are \[ Q_\Pi(C)=\int Q_\theta(C)\,\Pi(\mathrm{d}\theta),\qquad Q(C\mid X)=\int Q_\theta(C)\,\Pi(\mathrm{d}\theta\mid X). \]
Proposition. If \(Y\perp X\mid\theta\), then \(Q(C\mid X)=\mathbb P(Y\in C\mid X)\).
Proof. Under conditional independence, the joint law factors as \(P_\theta(\mathrm{d}x)Q_\theta(\mathrm{d}y)\Pi(\mathrm{d}\theta)\). Disintegrating first with respect to \(X\) leaves the conditional distribution of \(\theta\) given \(X\), and integrating \(Q_\theta(C)\) against this conditional distribution gives the displayed formula.
Marginal likelihood as a normalizing constant
The marginal likelihood is both a predictive density for the observed data and the denominator in Bayes formula.
Definition. In a dominated model, the marginal likelihood is \[ m(x)=\int_\Theta p_\theta(x)\,\Pi(\mathrm{d}\theta). \] For independent observations \(X_1,\ldots,X_n\), \(m_n(x^{(n)})=\int\prod_{i=1}^n p_\theta(x_i)\,\Pi(\mathrm{d}\theta)\).
Proposition. The sequential predictive factorization is \[ m_n(X_1,\ldots,X_n)=\prod_{i=1}^n p(X_i\mid X_1,\ldots,X_{i-1}), \] where the factor is the posterior predictive density based on the previous observations.
Proof. Write \(m_i(x_{1:i})/m_{i-1}(x_{1:i-1})\). Cancelling the common integral over the first \(i-1\) likelihood factors shows that this ratio is \(\int p_\theta(x_i)\Pi(\mathrm{d}\theta\mid x_{1:i-1})\). Multiplying the ratios gives the product formula.
Posterior predictive expectations
Predictive means are posterior means of conditional means, and predictive variances split into noise and parameter uncertainty.
Proposition. If \(Y\mid\theta\) has conditional mean \(a(\theta)\) and variance \(v(\theta)\), then \[ \mathbb E(Y\mid X)=\mathbb E(a(\theta)\mid X), \quad \operatorname{Var}(Y\mid X)=\mathbb E(v(\theta)\mid X)+\operatorname{Var}(a(\theta)\mid X). \]
Proof. These are the tower property and the conditional variance identity applied with the intermediate sigma-field generated by \(\theta\) and \(X\).
Model evidence and Occam factors
A model with a large parameter space is not automatically favored, because predictive mass is averaged over the prior rather than maximized.
Remark. The marginal likelihood rewards models that put high prior predictive mass near the observed data. Concentrated likelihood at a small prior volume can produce an Occam penalty, which is why Bayes factors differ from likelihood-ratio tests based only on maximized likelihoods.
Predictive factorization of evidence
Theorem. For conditionally independent observations, \[ m(x_{1:n}) =\prod_{i=1}^n p(x_i\mid x_{1:i-1}), \] where \[ p(x_i\mid x_{1:i-1}) =\int p_\theta(x_i) \Pi(\mathrm{d}\theta\mid x_{1:i-1}). \]
Proof. The ordinary chain rule gives \[ m(x_{1:n}) =\prod_i m(x_i\mid x_{1:i-1}). \] Conditional independence and posterior disintegration yield \[ m(x_i\mid x_{1:i-1}) =\int p_\theta(x_i)\Pi(\mathrm{d}\theta\mid x_{1:i-1}). \]
Marginal likelihood is therefore cumulative prequential performance under logarithmic score. Unlike leave-one-out prediction, each factor predicts a later observation from an earlier posterior and remains sensitive to the initial prior.
Beta-binomial prediction
Example. Under the beta-binomial model, \[ P(X_{n+1}=1\mid X_{1:n}) =E(\theta\mid X_{1:n}) =\frac{a+\sum_iX_i}{a+b+n}. \] For \(m\) future Bernoulli observations and future success count \(Y\), \[ P(Y=y\mid X_{1:n}) =\binom my \frac{B(a+x+y,b+n-x+m-y)} {B(a+x,b+n-x)}. \] This distribution is wider than a binomial with success probability equal to the posterior mean because it includes both future sampling variation and uncertainty about \(\theta\).
Posterior predictive variance decomposition
Proposition. For a scalar future observation \(\widetilde X\), \[ \operatorname{Var}(\widetilde X\mid x) =E\{\operatorname{Var}(\widetilde X\mid\theta)\mid x\} +\operatorname{Var}\{E(\widetilde X\mid\theta)\mid x\}. \]
Proof. Apply the conditional law of total variance with the posterior distribution of \(\theta\) as the mixing law.
The first term is irreducible future noise; the second is parameter uncertainty. Plug-in prediction at \(\widehat\theta\) omits the second term and is typically too concentrated.
Laplace expansion of the marginal likelihood
Theorem (regular Laplace approximation). Let \(\theta\in\mathbb R^d\), let \(\widehat\theta\) be an interior likelihood maximizer, and assume the observed information \[ J_n(\widehat\theta) =-\nabla^2\ell_n(\widehat\theta) \] is positive definite and of order \(n\). If the prior density \(\pi\) is continuous and positive near \(\widehat\theta\), then \[ m(x^{(n)}) =e^{\ell_n(\widehat\theta)} \pi(\widehat\theta) (2\pi)^{d/2} |J_n(\widehat\theta)|^{-1/2} \{1+o_P(1)\}. \]
Proof. Taylor-expand \[ \ell_n(\theta) =\ell_n(\widehat\theta) -\frac12(\theta-\widehat\theta)^\top J_n(\widehat\theta)(\theta-\widehat\theta) +r_n(\theta). \] Localization makes the integral outside an \(n^{-1/2}\)-neighborhood negligible, and the remainder is uniform inside it. Replace \(\pi(\theta)\) by \(\pi(\widehat\theta)\) and integrate the Gaussian kernel.
Corollary. If \(J_n(\widehat\theta)=nI(\theta_0)+o_P(n)\), then \[ \log m(x^{(n)}) =\ell_n(\widehat\theta) -\frac d2\log n +\log\pi(\widehat\theta) +\frac d2\log(2\pi) -\frac12\log|I(\theta_0)| +o_P(1). \] The term \(-(d/2)\log n\) is the regular Occam penalty underlying BIC.
Prediction under logarithmic score
Proposition. If the true predictive density is \(q\), the expected logarithmic score \[ E_q\{\log p(X)\} \] is uniquely maximized at \(p=q\), up to \(q\)-null sets.
Proof. \[ E_q\log q(X)-E_q\log p(X) =\operatorname{KL}(q\|p)\ge0, \] with equality exactly when \(p=q\) almost everywhere.
This strict propriety explains why posterior predictive distributions should be assessed as distributions, not only through point predictions. It also clarifies the difference between evidence, which scores sequential prior prediction, and cross-validation, which estimates prediction after fitting to nearly all the observed data.