Modern Bayesian procedures often combine sparsity, structure, and variational approximation. The mathematical question is whether computationally tractable posteriors still concentrate near the truth with calibrated uncertainty.
Sparse priors
Sparsity priors put most mass on low-dimensional structures inside a high-dimensional space.
Definition. A spike-and-slab prior for \(\theta\in\mathbb R^p\) has indicators \(z_j\sim\operatorname{Bernoulli}(q)\) and \[ \theta_j\mid z_j\sim (1-z_j)\delta_0+z_j G. \]
Proposition. The prior mass of a support \(S\) is \(q^{|S|}(1-q)^{p-|S|}\), before accounting for slab density on the nonzero coefficients.
Proof. The indicators are independent Bernoulli variables, so the probability of exactly the support \(S\) is the product of \(q\) over \(S\) and \(1-q\) over its complement.
Posterior contraction in sparse models
Rates depend on the true sparsity and the logarithmic cost of searching over supports.
Theorem. In many sparse normal means or regression problems, the benchmark squared-error rate is of order \[ s_0\log(p/s_0)/n \] for \(s_0\) nonzero coefficients.
Proof. The variance cost of estimating \(s_0\) coefficients is proportional to \(s_0/n\), and the model-selection cost is the logarithm of the number of possible supports, \(\log {p\choose s_0}\asymp s_0\log(p/s_0)\), divided by \(n\).
Variational sparse Bayes
Mean-field approximations often factorize over inclusion indicators and coefficients.
Definition. A structured variational family may use \[ q(z,\theta)=\prod_{j=1}^p q_j(z_j,\theta_j), \] or richer block factorizations that preserve known dependence.
Remark. A too-factorized family can underestimate posterior dependence and uncertainty, but may still recover support or prediction risk under suitable beta-min and design conditions.
Computation and theory
The ELBO is an optimization objective, while contraction is a statistical property. Both must be checked.
Proposition. If a variational distribution is close in KL to a posterior that contracts at rate \(\varepsilon_n\), then bounded posterior functionals are close up to the variational approximation error.
Proof. Pinsker's inequality converts KL error into total variation error. Total variation controls bounded expectation differences, while posterior contraction controls the target posterior mass outside the \(\varepsilon_n\)-ball.
Spike-and-slab variational families
Definition. A coordinatewise spike-and-slab variational distribution has \[ q(\beta,z) =\prod_{j=1}^p \left[ \gamma_jq_j(\beta_j)1\{z_j=1\} +(1-\gamma_j)\delta_0(\beta_j)1\{z_j=0\} \right], \] where \(\gamma_j\in[0,1]\) approximates posterior inclusion probability. Structured variants retain dependence within groups or through a sparse covariance graph.
Proposition. The expected variational model size is \[ E_q|S|=\sum_{j=1}^p\gamma_j, \] and its entropy contains the Bernoulli term \[ -\sum_j\{ \gamma_j\log\gamma_j+ (1-\gamma_j)\log(1-\gamma_j) \} \] plus conditional slab entropies.
Proof. Since \(|S|=\sum_jz_j\), linearity gives the first identity. Entropy of a product is the sum of factor entropies, and each mixed discrete-continuous factor decomposes into Bernoulli entropy plus \(\gamma_j\) times slab entropy.
Variational contraction from a comparison distribution
Theorem (oracle variational bound). Let \(\widehat Q\) minimize \[ \operatorname{KL}\{Q\|\Pi(\cdot\mid X^{(n)})\} \] over \(\mathcal Q\). Suppose there exists \(Q^\star\in\mathcal Q\), concentrated in a \(d\)-ball of radius \(\varepsilon_n\) around truth, such that \[ \operatorname{KL}(Q^\star\|\Pi) +E_{Q^\star}K_n(P_0,P_\theta) \le Cn\varepsilon_n^2. \] Under an exponential testing condition for loss \(d^2\), one obtains \[ E_0E_{\widehat Q} d^2(P_\theta,P_0) \lesssim\varepsilon_n^2 \] up to constants and testing remainders.
Proof outline. The variational optimum is no worse in posterior KL than \(Q^\star\). Use the ELBO identity to express this comparison as empirical loss plus prior KL. An exponential test or PAC-Bayes change-of-measure inequality converts the empirical criterion into expected statistical loss uniformly over \(Q\). The comparison-distribution bound yields the rate.
The theorem does not require \(\widehat Q\) to be close to the exact posterior in total variation. It proves statistical accuracy directly through one well-chosen variational approximation to truth.
Sparse linear-regression rate
Theorem (schematic). In \[ Y=X\beta_0+\varepsilon,\qquad \varepsilon\sim N(0,\sigma^2I_n), \] suppose \(\beta_0\) is \(s_0\)-sparse, the design satisfies a sparse compatibility condition, and a spike-and-slab variational family uses a complexity prior and slabs with sufficient local mass. Then \[ E_{\widehat Q} \frac{\|X(\beta-\beta_0)\|_2^2}{n} =O_{P_0}\left( \frac{s_0\log p}{n} \right), \] and typically \[ E_{\widehat Q}\|\beta-\beta_0\|_2^2 =O_{P_0}\left( \frac{s_0\log p}{n} \right) \] under restricted eigenvalues.
Proof outline. Choose \(Q^\star\) supported on the true model, with slab factors in balls of radius \(n^{-1/2}\) around active coefficients. Prior and entropy cost is \[ O\{s_0\log p\}, \] matching the sparse model count. Gaussian likelihood KL is proportional to prediction loss. Apply the oracle variational theorem, then restricted eigenvalues convert prediction to coefficient loss.
Selection and thresholding
Remark. Rate-optimal prediction does not imply \[ \gamma_j\to1\quad(j\in S_0), \qquad \gamma_j\to0\quad(j\notin S_0). \] Selection additionally requires beta-min separation, design incoherence, and enough variational flexibility to distinguish correlated alternatives. Thresholding at \(\gamma_j>1/2\) is a decision rule whose loss should be specified.
Mean-field underdispersion in sparse models
Even if support probabilities are accurate, factorized slabs omit posterior dependence within a selected model. For a linear functional \(a^\top\beta\), \[ \operatorname{Var}_q(a^\top\beta) =\sum_ja_j^2\operatorname{Var}_q(\beta_j) \] omits \[ 2\sum_{j<k}a_ja_k \operatorname{Cov}_\Pi(\beta_j,\beta_k\mid X). \] Correlated designs can make this omission first order. Richer low-rank covariance factors or post-selection Gaussian correction can restore local dependence.
Optimization stationarity is not statistical validity
Coordinate-ascent fixed points can be local optima. Let \[ \mathcal L^\star=\sup_{Q\in\mathcal Q}\mathcal L(Q), \qquad \delta_{\mathrm{opt}} =\mathcal L^\star-\mathcal L(\widehat Q_{\mathrm{alg}}). \] A statistical theorem for the global variational optimum extends to the algorithmic output only if \[ \delta_{\mathrm{opt}} =o_P(n\varepsilon_n^2) \] in the scale used by the oracle inequality. Initialization sensitivity and multiple restarts are therefore part of the inferential error analysis.
Structured sparsity
Example (group spike-and-slab). Partition coefficients into groups \(G_1,\ldots,G_m\), introduce \[ z_g\sim\operatorname{Bernoulli}(\gamma_g), \] and conditionally use a multivariate slab on \(\beta_{G_g}\). The complexity rate becomes \[ \frac{s_G\log m+\sum_{g\in S_G}|G_g|}{n}, \] where \(s_G\) is the number of active groups. A coordinatewise variational family cannot represent all-or-none group inclusion without additional coupling.
Calibration strategies
Possible corrections include linear-response covariance, importance weighting of variational draws, refitting an exact sampler on selected coordinates, bootstrap calibration, and one-step debiasing of a target functional. Each addresses a different discrepancy. A correction should be validated for the reported functional, not justified by a generic claim that variational Bayes is “close.”