Bayesian model comparison can be based on marginal likelihood, predictive performance, or decision loss. These criteria answer different questions and should not be conflated.
Bayes factors and posterior model odds
Bayes factors compare models through their prior predictive densities.
Definition. For models \(M_1,M_2\), the Bayes factor in favor of \(M_1\) over \(M_2\) is \[ B_{12}=\frac{m_1(X)}{m_2(X)}. \] Posterior odds satisfy \[ \frac{\Pi(M_1\mid X)}{\Pi(M_2\mid X)} =B_{12}\frac{\Pi(M_1)}{\Pi(M_2)}. \]
Proof. Apply Bayes formula on the discrete model index. The likelihood for model \(M_j\) after integrating its parameters is precisely \(m_j(X)\).
Laplace approximation and the Occam factor
Marginal likelihood automatically penalizes unused parameter volume.
Proposition. If a \(d\)-dimensional regular model has posterior mode \(\hat\theta\), then formally \[ m(X)\approx p_{\hat\theta}(X)\pi(\hat\theta)(2\pi)^{d/2}\det\{-\nabla^2\log[p_\theta(X)\pi(\theta)]_{\hat\theta}\}^{-1/2}. \]
Proof. Second-order Taylor expansion of the log integrand at \(\hat\theta\) gives a Gaussian integral. The determinant term is the local posterior volume and produces the Occam factor.
Cross-validation and predictive scoring
Cross-validation evaluates predictive accuracy under a chosen scoring rule rather than prior predictive evidence.
Definition. Leave-one-out expected log predictive density estimates \[ \sum_{i=1}^n \log p(y_i\mid y_{-i}) \] where \(p(y_i\mid y_{-i})\) is the posterior predictive density fitted without observation \(i\).
Remark. Bayes factors are sensitive to prior mass on parameters; cross-validation is usually less sensitive to diffuse prior volume but targets predictive performance rather than posterior model probability.
Stacking and model averaging
When no model is true, averaging for prediction can be preferable to selecting one model.
Proposition. Stacking chooses weights to maximize a cross-validated predictive score over convex combinations of predictive distributions.
Proof. The criterion is a concave optimization problem in the weights for logarithmic score because the log of a positive affine combination is concave. The resulting weights target predictive performance rather than posterior model probabilities.
Bayes factors and prior sensitivity
Definition. For models \(M_0,M_1\), \[ BF_{10} =\frac{m_1(y)}{m_0(y)} =\frac{\int p_1(y\mid\theta_1) \pi_1(\mathrm{d}\theta_1)} {\int p_0(y\mid\theta_0) \pi_0(\mathrm{d}\theta_0)}. \] Posterior model odds satisfy \[ \frac{P(M_1\mid y)}{P(M_0\mid y)} =BF_{10}\frac{P(M_1)}{P(M_0)}. \]
Theorem (regular Bayes-factor expansion). If model \(M_k\) has regular dimension \(d_k\), then \[ 2\log BF_{10} =2\{\ell_1(\widehat\theta_1)-\ell_0(\widehat\theta_0)\} -(d_1-d_0)\log n +O_P(1), \] where the \(O_P(1)\) term contains prior densities and information determinants.
Proof. Apply the Laplace expansion \[ \log m_k(y) =\ell_k(\widehat\theta_k) -\frac{d_k}{2}\log n+O_P(1) \] to each model and subtract.
The \(O_P(1)\) prior term can matter greatly in finite samples, especially for weakly identified parameters. Bayes-factor robustness must be assessed by varying scientifically plausible proper priors, not by quoting only a BIC approximation.
Jeffreys-Lindley phenomenon
Example. Let \[ \bar X\mid\theta\sim N(\theta,1/n). \] Compare \(H_0:\theta=0\) with \(H_1:\theta\sim N(0,\tau^2)\). Then \[ BF_{01} =\sqrt{1+n\tau^2} \exp\left[ -\frac{n^2\tau^2\bar X^2} {2(1+n\tau^2)} \right]. \] If \(z=\sqrt n\,\bar X\) is held fixed while \(n\to\infty\), then \[ BF_{01}\sim\sqrt n\,\tau\,e^{-z^2/2}\to\infty. \] Thus a fixed frequentist \(z\)-value can coexist with increasing posterior support for the point null.
Derivation. Under \(H_0\), \(\bar X\sim N(0,1/n)\); under \(H_1\), integrating \(\theta\) gives \(\bar X\sim N(0,\tau^2+1/n)\). Taking the ratio of these two normal densities gives the formula.
Expected log predictive density
Definition. Leave-one-out expected log predictive density is estimated by \[ \widehat{\operatorname{elpd}}_{\mathrm{loo}} =\sum_{i=1}^n \log p(y_i\mid y_{-i}), \] where \[ p(y_i\mid y_{-i}) =\int p(y_i\mid\theta) \Pi(\mathrm{d}\theta\mid y_{-i}). \]
Proposition. Under exchangeable sampling, the expectation of one leave-one-out term equals the expected log score of a posterior predictive distribution trained on \(n-1\) fresh observations.
Proof. By exchangeability, for each \(i\), the pair \((Y_i,Y_{-i})\) has the same law as a fresh test observation and an independent training sample of size \(n-1\), after relabeling. Taking expectations gives the claim.
Importance approximation and Pareto diagnostics
From full posterior draws \(\theta^{(s)}\sim\Pi(\cdot\mid y)\), \[ p(y_i\mid y_{-i}) =\left[ E_{\Pi(\cdot\mid y)} \left\{\frac1{p(y_i\mid\theta)}\right\} \right]^{-1}. \] The raw importance ratios \[ r_i^{(s)}=\frac1{p(y_i\mid\theta^{(s)})} \] can have heavy tails. Pareto-smoothed importance sampling regularizes their largest values and estimates a generalized-Pareto tail shape \(k\). Large \(k\) signals that deleting observation \(i\) moves the posterior too far for stable importance reweighting; exact refitting or \(K\)-fold cross-validation is then safer.
WAIC and effective complexity
Definition. The widely applicable information criterion uses \[ \operatorname{lppd} =\sum_i\log E_{\Pi(\cdot\mid y)} p(y_i\mid\theta), \] \[ p_{\mathrm{WAIC}} =\sum_i\operatorname{Var}_{\Pi(\cdot\mid y)} \{\log p(y_i\mid\theta)\}, \] and \[ \widehat{\operatorname{elpd}}_{\mathrm{WAIC}} =\operatorname{lppd}-p_{\mathrm{WAIC}}. \]
WAIC is asymptotically related to leave-one-out prediction, including in some singular models, but its finite-sample variance expansion can fail with influential observations. Pointwise diagnostics and LOO are usually more informative than one aggregate criterion.
Stacking predictive distributions
Theorem. Given leave-one-out predictive densities \(p_{k,-i}(y_i)\), stacking solves \[ \widehat w \in\arg\max_{w\in\Delta_{K-1}} \sum_{i=1}^n \log\left\{ \sum_{k=1}^Kw_kp_{k,-i}(y_i) \right\}. \] The objective is concave in \(w\); it is strictly concave when the predictive vectors distinguish every feasible direction.
Proof. Each summand is the logarithm of a positive affine function of \(w\), hence concave. Its Hessian is a negative semidefinite rank-one matrix. Summing gives strict negativity on directions not annihilated by all predictive vectors.
Posterior model probabilities estimate belief in a model index under an \(M\)-closed assumption that one listed model generated the data. Stacking chooses a predictive mixture under an \(M\)-open view in which every listed model may be wrong. These are different inferential goals, so their weights need not agree.