Variational Bayes is best read as projection of the posterior onto a tractable family. The chosen KL direction determines the geometry and the typical bias of the approximation.
Evidence lower bound
The optimization objective is an exact decomposition of the marginal likelihood.
Definition. For a density \(q\) on \(\Theta\), the evidence lower bound is \[ \mathcal L(q)=\int q(\theta)\log\frac{p_\theta(X)\pi(\theta)}{q(\theta)}\,\mathrm{d}\theta. \]
Proposition. The identity \[ \log m(X)=\mathcal L(q)+\mathrm{KL}(q\,\|\,\Pi(\cdot\mid X)) \] holds whenever the terms are well-defined.
Proof. Use \(\pi(\theta\mid X)=p_\theta(X)\pi(\theta)/m(X)\). Then \[ \mathrm{KL}(q\,\|\,\Pi)=\int q\log\frac{q\,m(X)}{p_\theta(X)\pi(\theta)} =\log m(X)-\mathcal L(q). \]
Mean-field coordinate updates
Mean-field VB turns the variational problem into a sequence of conditional expectation calculations.
Theorem. If \(q(\theta)=\prod_{j=1}^k q_j(\theta_j)\), then the coordinate optimizer satisfies \[ \log q_j^*(\theta_j)=\mathbb E_{-j}\log\{p_\theta(X)\pi(\theta)\}+\mathrm{constant}. \]
Proof. Hold all other factors fixed and keep only terms depending on \(q_j\). The objective is a linear functional of \(q_j\) minus \(\int q_j\log q_j\), subject to normalization. The Euler equation gives the displayed exponential form.
Reverse and forward KL
The direction of KL controls whether the approximation is mode-seeking or mass-covering.
Definition. Reverse KL is \(\mathrm{KL}(q\|\pi)=\int q\log(q/\pi)\). Forward KL is \(\mathrm{KL}(\pi\|q)=\int \pi\log(\pi/q)\).
Remark. Reverse KL heavily penalizes putting mass where the posterior density is small, so it can underestimate uncertainty in multimodal or skewed problems. Forward KL penalizes missing posterior mass but is often harder to optimize.
Variational uncertainty
A good ELBO value does not automatically imply accurate credible sets.
Proposition. If the variational family excludes posterior dependence, then marginal variances can be systematically underestimated even when posterior means are accurate.
Proof. Dependence contributes covariance terms to uncertainty in linear functionals. A product approximation sets cross-covariances to zero, so the induced variance of sums or contrasts may be too small unless the missing covariance happens to vanish.
ELBO decomposition
Theorem. Let \[ \pi(\theta\mid x) =\frac{p(x,\theta)}{p(x)} \] and let \(q\) be any density dominated by the posterior. Then \[ \log p(x) =\mathcal L(q) +\operatorname{KL}\{q\|\pi(\cdot\mid x)\}, \] where \[ \mathcal L(q) =E_q\log p(x,\theta)-E_q\log q(\theta) \] is the evidence lower bound.
Proof. \[ \operatorname{KL}(q\|\pi) =E_q\log q-E_q\log p(x,\theta)+\log p(x). \] Rearrange. Nonnegativity of KL gives \(\mathcal L(q)\le\log p(x)\).
Mean-field coordinate updates
Theorem. For a product family \[ q(\theta)=\prod_{j=1}^mq_j(\theta_j), \] holding \(q_{-j}\) fixed, the ELBO-maximizing factor satisfies \[ \log q_j^\star(\theta_j) =E_{q_{-j}}\log p(x,\theta)+C_j. \]
Proof. Terms depending on \(q_j\) are \[ \int q_j(\theta_j) E_{q_{-j}}\log p(x,\theta)\,\mathrm{d}\theta_j -\int q_j\log q_j. \] Adding a Lagrange multiplier for normalization and taking the functional derivative gives \[ E_{q_{-j}}\log p-\log q_j-1+\lambda=0. \] Exponentiation and normalization yield the update.
Proposition. Exact coordinate updates never decrease the ELBO, and every limit point satisfying regularity is coordinatewise stationary. They need not reach the global optimum because the product-family ELBO is generally nonconcave jointly in all factors.
A Gaussian mean-field calculation
Example. Let the target be \[ \pi(\theta)=N_2(0,\Sigma),\qquad \Sigma= \begin{bmatrix}1&\rho\\\rho&1\end{bmatrix}. \] The reverse-KL optimal product Gaussian \[ q=N(m_1,v_1)\otimes N(m_2,v_2) \] has \[ m_1=m_2=0,\qquad v_1=v_2=1-\rho^2. \] Thus each variational marginal variance is smaller than the true marginal variance \(1\).
Proof. The target precision is \[ \Sigma^{-1} =\frac1{1-\rho^2} \begin{bmatrix}1&-\rho\\-\rho&1\end{bmatrix}. \] The mean-field coordinate update for \(\theta_1\) has precision \((\Sigma^{-1})_{11}=1/(1-\rho^2)\) and mean \(\rho E_q\theta_2\); symmetrically for \(\theta_2\). The fixed point has zero means and variance \(1-\rho^2\).
Reverse and forward KL geometry
Example (separated mixture). Let \[ \pi=\frac12N(-a,1)+\frac12N(a,1) \] with large \(a\), and approximate by one Gaussian.
- Minimizing \(\operatorname{KL}(q\|\pi)\) can choose one mode, because placing mass in the low-density valley is heavily penalized while omitting another target mode is not directly penalized under expectation \(q\).
- Minimizing \(\operatorname{KL}(\pi\|q)\) matches target moments, giving approximately \(N(0,1+a^2)\), which covers both modes and the low-density valley.
“Mode seeking” and “mass covering” are useful tendencies, not universal theorems for every variational family. The direction of KL determines which distribution supplies the expectation and therefore which support errors are costly.
Stochastic gradients and reparameterization
Theorem (pathwise gradient). Suppose \[ \theta=T_\lambda(\varepsilon), \qquad \varepsilon\sim r, \] and differentiation may pass under the integral. Then \[ \nabla_\lambda E_{q_\lambda}f(\theta,\lambda) =E_r\left[ \nabla_\theta f\{T_\lambda(\varepsilon),\lambda\} \nabla_\lambda T_\lambda(\varepsilon) +\nabla_\lambda f\{T_\lambda(\varepsilon),\lambda\} \right]. \]
Proof. Rewrite the expectation as \[ \int f\{T_\lambda(\varepsilon),\lambda\}r(\varepsilon) \,\mathrm{d}\varepsilon. \] Differentiate under the integral and apply the chain rule.
For a location-scale Gaussian, \(\theta=\mu+L\varepsilon\), \(\varepsilon\sim N(0,I)\). Pathwise gradients usually have lower variance than score-function gradients because they use local derivatives of the log joint, but they require a differentiable simulator.
Variational uncertainty under regular asymptotics
Proposition. Suppose the exact local posterior is approximately \[ N_d(\widehat\theta,(nI)^{-1}). \] The reverse-KL projection onto diagonal Gaussians has covariance \[ \frac1n \operatorname{diag}\left( \frac1{I_{11}},\ldots,\frac1{I_{dd}} \right), \] not the diagonal of \(I^{-1}/n\) in general.
Proof. Repeat the Gaussian mean-field calculation with target precision \(nI\). Each factor's optimal precision is the corresponding diagonal precision \(nI_{jj}\).
When parameters are correlated, \(1/I_{jj}\le(I^{-1})_{jj}\), so credible intervals are too narrow. Linear-response corrections, importance weighting, richer covariance families, or calibrated bootstrap methods can improve uncertainty, but each introduces additional assumptions and computation.
Three separable errors
A practical variational result contains:
- statistical error: the exact posterior differs from the data-generating target;
- approximation error: the best member of \(\mathcal Q\) differs from the exact posterior;
- optimization error: the algorithm has not reached the best member of \(\mathcal Q\).
An ELBO trace diagnoses only the third. Enlarging \(\mathcal Q\) may reduce the second while worsening the optimization landscape. Theoretical guarantees should state which of these errors they control and in which metric.