Redundancy problem
When predictors are strongly correlated, many models represent nearly the same predictive information. Ordinary independent model priors can overcount a scientific idea just because it has many redundant encodings.
Definition. A dilution prior assigns smaller total prior mass to clusters of similar models. A determinantal point process prior favors diverse subsets through probabilities proportional to \(\det(K_S)\), where \(K_S\) is a similarity kernel submatrix.
Proposition. If two predictors are nearly identical, a DPP prior with similarity kernel \(K\) assigns small prior mass to including both because \(\det(K_{\{j,k\}})=K_{jj}K_{kk}-K_{jk}^2\) is small when \(K_{jk}^2\approx K_{jj}K_{kk}\).
Proof. For a two-by-two kernel submatrix, compute the determinant. Near collinearity makes the two columns nearly linearly dependent, and determinant measures squared volume.
Projection predictive inference
Definition. Projection predictive inference chooses a smaller model by minimizing predictive discrepancy from a reference posterior, for example \[ \min_{\theta_S}D\{p(\tilde y\mid x,\theta_{\mathrm{ref}})\|p_S(\tilde y\mid x_S,\theta_S)\}. \]
Remark. Projection targets predictive preservation, not exact recovery of the true support. This is essential when redundant predictors make exact support scientifically unstable.
Volume interpretation of DPP priors
Proposition. If columns indexed by \(S\) are represented by feature vectors \(v_j\), then \(\det(V_S^\top V_S)\) is the squared volume of the parallelepiped spanned by those vectors. A DPP prior therefore penalizes nearly collinear sets.
Proof. The Gram determinant formula states that the squared \(s\)-dimensional volume of \(v_{j_1},\ldots,v_{j_s}\) is \(\det(V_S^\top V_S)\). Near linear dependence makes this determinant small.
Projection predictive target
Remark. Projection predictive inference answers a different question from selection consistency: find a small model that preserves the reference posterior predictive distribution. This is often the scientifically stable question under collinearity.
Equivalence classes of redundant models
Definition. Let \(d(M,M')\) measure predictive or column-space distance between models. A dilution prior seeks approximately \[ \sum_{M'\in B(M,\varepsilon)} \Pi(M') \] to remain stable when a model representation is duplicated by nearly equivalent predictors.
Uniform model priors are uniform over syntactic subsets, not predictive directions. Duplicating one predictor changes posterior prior odds even though the span of possible fitted values changes little.
Determinantal point-process model prior
Definition. Let \(L\succeq0\) be a \(p\times p\) similarity-quality matrix. An \(L\)-ensemble determinantal point process assigns \[ \Pi(S)=\frac{\det(L_S)}{\det(I+L)}, \qquad S\subseteq[p]. \]
Theorem (normalization). \[ \sum_{S\subseteq[p]}\det(L_S) =\det(I+L). \]
Proof. Diagonalize \[ L=U\operatorname{diag}(\lambda_1,\ldots,\lambda_p)U^\top. \] The sum of all principal minors of size \(k\) is the \(k\)-th elementary symmetric polynomial in the eigenvalues. Summing over \(k\) gives \[ \prod_{j=1}^p(1+\lambda_j)=\det(I+L). \]
Proposition. If \[ L_{ij}=q_iq_jv_i^\top v_j \] with normalized feature vectors \(v_j\), then \[ \det(L_S) =\left(\prod_{j\in S}q_j^2\right) \det(V_S^\top V_S). \] The first factor rewards individual quality; the Gram determinant penalizes collinearity.
Fixed-size DPP and model dimension
A \(k\)-DPP conditions on \(|S|=k\): \[ \Pi(S\mid|S|=k) \propto\det(L_S). \] A separate prior on \(k\) controls model-size complexity. Without it, determinant repulsion alone may still favor overly large or small subsets depending on \(L\)'s spectrum.
Projection predictive objective
Definition. Let a reference posterior draw \(\theta^\star\) define predictive density \[ p^\star(\widetilde y\mid x,\theta^\star). \] For submodel \(S\), its projected parameter solves \[ \theta_S^\perp(\theta^\star) \in\arg\min_{\theta_S} E_{x\sim\nu} \operatorname{KL}\left\{ p^\star(\cdot\mid x,\theta^\star) \|p_S(\cdot\mid x,\theta_S) \right\}. \]
Theorem (Gaussian linear projection). Suppose the reference predictive mean is \[ \mu^\star=X\beta^\star \] and the submodel uses \(X_S\beta_S\) with common Gaussian noise variance. Then \[ \beta_S^\perp =(X_S^\top X_S)^\dagger X_S^\top X\beta^\star, \] and the mean-prediction KL loss is proportional to \[ \|(I-P_S)X\beta^\star\|_2^2, \qquad P_S=X_S(X_S^\top X_S)^\dagger X_S^\top. \]
Proof. KL divergence between normals with common variance is proportional to squared mean difference: \[ \|X\beta^\star-X_S\beta_S\|_2^2. \] Least squares gives the pseudoinverse projection, and the residual is the orthogonal complement of the column space of \(X_S\).
For GLMs, projection minimizes expected negative log likelihood under reference predictive draws, usually by iteratively reweighted fitting for every reference draw or cluster of draws.
Forward projection search
Definition. At each step, add the variable that minimizes cross-validated predictive loss after projection. Stop when the submodel's utility is within a prespecified tolerance of the reference model.
The tolerance has a direct predictive interpretation; it is not a posterior probability that omitted coefficients are zero. Search-induced optimism must be corrected by nested cross-validation or by evaluating the complete projection-selection process on held-out folds.
Reference-model uncertainty
Proposition. Projecting each reference posterior draw and retaining the collection \[ \{\theta_S^\perp(\theta^{\star(s)})\}_{s=1}^N \] propagates uncertainty from the reference predictive distribution into the submodel parameters. Projecting only the reference posterior mean generally understates this uncertainty because projection is nonlinear.
Proof. For nonlinear map \(T\), \[ T(E\theta^\star)\ne E\{T(\theta^\star)\} \] in general, and the single value \(T(E\theta^\star)\) has no between-draw variability. Applying \(T\) drawwise approximates the pushforward posterior.
DPP and projection answer different stages
A DPP is a prior over candidate models that discourages redundant subsets before or during Bayesian model averaging. Projection predictive inference starts from a fitted reference predictive distribution and seeks a compact surrogate afterward. They can be combined, but determinant prior mass should not be interpreted as projected predictive fidelity, and projection loss should not be interpreted as posterior model probability.