Entropic regularization
Entropic VI is useful when the goal is not only tractability but also preserving dependence or spreading mass. The mathematical object is a tradeoff between fidelity to the target and entropy of the approximation.
Definition. An entropically regularized variational problem has the form \[ \min_{Q\in\mathcal Q}\; E_Q[-\log p(X,\theta)]+\rho D(Q\|R)-\tau H(Q), \] where \(R\) is a reference measure and \(H(Q)=-E_Q\log q\) when \(Q\) has density \(q\).
Proposition. If \(\mathcal Q\) is unrestricted and the objective is \(D(Q\|\Pi)-\tau H(Q)\) relative to a base measure, the optimizer is a tempered density proportional to \(\pi^{1/(1+\tau)}\).
Proof. Write the objective as \(\int q\log(q/\pi)\,\mathrm{d}\mu+\tau\int q\log q\,\mathrm{d}\mu\). Taking the variational derivative under \(\int q=1\) gives \((1+\tau)\log q-\log\pi+\mathrm{constant}=0\), hence \(q\propto \pi^{1/(1+\tau)}\).
Mirror descent
Remark. Entropy is the Bregman generator behind exponentiated-gradient and mirror-descent updates. This is the optimization bridge between entropic VI, natural gradient methods, and information geometry.
KL-proximal update
Theorem. A KL-proximal step for minimizing a linearized objective \(E_Q g\) over probability measures satisfies \[ q^+(\theta)\propto q(\theta)\exp\{-\eta g(\theta)\}. \]
Proof. Minimize \(E_Q g+\eta^{-1}\operatorname{KL}(Q\|Q^{old})\) subject to \(\int q=1\). The Euler equation gives \(g+\eta^{-1}(\log q-\log q^{old}+1)+\lambda=0\), which yields the exponential tilt.
Temperature and uncertainty
Remark. Entropic smoothing can prevent premature collapse of the variational approximation, but it also changes the target unless the temperature is sent to zero or corrected by a final projection.
Gibbs variational principle
Theorem. Let \(P\) be a reference measure, let \(g\) be measurable, and let \(\eta>0\). Then \[ \inf_{Q\ll P} \left\{ E_Qg+\frac1\eta\operatorname{KL}(Q\|P) \right\} =-\frac1\eta \log E_Pe^{-\eta g}. \] The unique optimizer is \[ \frac{\mathrm{d}Q^\star}{\mathrm{d}P} =\frac{e^{-\eta g}} {E_Pe^{-\eta g}}. \]
Proof. Define \(Q^\star\) by the exponential tilt. Then \[ \operatorname{KL}(Q\|Q^\star) =\operatorname{KL}(Q\|P) +\eta E_Qg +\log E_Pe^{-\eta g}. \] Nonnegativity gives the lower bound and equality at \(Q^\star\).
Ordinary Bayes is the special case in which \(P\) is the prior, \(g\) is negative average log likelihood, and \(\eta\) is sample size under the chosen normalization.
KL-proximal mirror descent
Theorem. Given current distribution \(Q_k\) and functional gradient \(g_k\), the update \[ Q_{k+1} \in\arg\min_Q \left\{ E_Qg_k+\frac1{\eta_k} \operatorname{KL}(Q\|Q_k) \right\} \] has density \[ q_{k+1}(\theta) =\frac{ q_k(\theta)e^{-\eta_kg_k(\theta)} }{ \int q_k(u)e^{-\eta_kg_k(u)}\,\mathrm{d}u }. \]
Proof. Apply the Gibbs variational principle with reference measure \(Q_k\).
The KL term is the Bregman divergence generated by negative entropy. It keeps support inside that of \(Q_k\); if \(q_k\) assigns zero mass to a region, this mirror step cannot create it.
Finite-dimensional exponentiated gradient
Example. On a simplex \[ \Delta_{m-1}=\{w_j\ge0,\sum_jw_j=1\}, \] the entropic mirror update for gradient vector \(g_k\) is \[ w_{k+1,j} =\frac{w_{k,j}e^{-\eta_kg_{k,j}}} {\sum_\ell w_{k,\ell}e^{-\eta_kg_{k,\ell}}}. \] For small \(\eta_k\), \[ w_{k+1,j}-w_{k,j} \approx-\eta_kw_{k,j} \{g_{k,j}-\bar g_k\}, \] the replicator equation.
Regret bound
Theorem. Suppose losses \(g_k\in[0,1]^m\), start from uniform weights, and use constant \(\eta>0\). Exponentiated gradient satisfies, for every \(u\in\Delta_{m-1}\), \[ \sum_{k=1}^T \langle w_k-u,g_k\rangle \le\frac{\log m}{\eta} +\frac{\eta T}{8}. \] Choosing \(\eta=\sqrt{8\log m/T}\) gives regret \[ O(\sqrt{T\log m}). \]
Proof. Track \(\operatorname{KL}(u\|w_k)\). The update and Hoeffding's lemma give \[ \operatorname{KL}(u\|w_{k+1}) -\operatorname{KL}(u\|w_k) \le \eta\langle u-w_k,g_k\rangle +\frac{\eta^2}{8}. \] Sum over \(k\), use nonnegativity of final KL and \(\operatorname{KL}(u\|w_1)\le\log m\), then divide by \(\eta\).
Tempered posteriors
Definition. A fractional posterior is \[ \Pi_\alpha(\mathrm{d}\theta\mid X) \propto p_\theta(X)^\alpha \Pi(\mathrm{d}\theta), \qquad 0<\alpha<1. \] In a regular model its local covariance is approximately \[ \frac1{\alpha n}I^{-1}. \]
Derivation. Multiplying the LAN log likelihood by \(\alpha\) changes the quadratic term from \(-h^\top Ih/2\) to \(-\alpha h^\top Ih/2\), so the local Gaussian precision is \(\alpha I\).
Fractional posteriors can require weaker prior-mass conditions and offer robustness under misspecification, but their credible sets are wider by \(1/\sqrt\alpha\) under correct specification and are not automatically calibrated under misspecification.
Entropy-regularized optimization over measures
For a functional \[ \mathcal F(Q)=E_QV+\operatorname{KL}(Q\|P), \] the minimizer is a Gibbs measure. Gradient flows of \(\mathcal F\) in different geometries produce different algorithms: KL mirror descent updates density multiplicatively, while Wasserstein gradient flow yields Langevin-type transport. “Entropic variational inference” should specify the ambient geometry, reference measure, and temperature.
Annealing and path sampling
Proposition. For power posteriors \[ \pi_t(\theta) \propto p(X\mid\theta)^t\pi(\theta), \qquad t\in[0,1], \] the log normalizing constant satisfies \[ \log p(X) =\int_0^1 E_{\pi_t}\log p(X\mid\theta)\,\mathrm{d}t. \]
Proof. Let \[ Z(t)=\int p(X\mid\theta)^t\pi(\theta)\,\mathrm{d}\theta. \] Differentiate: \[ \frac{\mathrm{d}}{\mathrm{d}t}\log Z(t) =E_{\pi_t}\log p(X\mid\theta). \] Integrate from \(0\), where \(Z(0)=1\), to \(1\), where \(Z(1)=p(X)\).
Annealing can bridge separated distributions, but a schedule with too few temperatures creates weight degeneracy or mode loss. Effective sample size across temperature steps is a computational diagnostic, not proof that all modes were found.