Entropy supplies a mirror geometry directly on probability measures. In finite dimensions it produces exponentiated gradients; in variational inference it produces Gibbs tilts and KL-proximal updates of densities.
The Gibbs variational principle
Log-partition duality. Let \(\mu\) be a probability measure and let \(h\) satisfy \(0<\int e^h\,\mathrm{d}\mu<\infty\). Then \[ \log\int e^h\,\mathrm{d}\mu = \sup_{q\ll\mu} \left\{\int h\,\mathrm{d}q-\operatorname{KL}(q\|\mu)\right\}. \] The unique maximizing probability measure has density \[ \frac{\mathrm{d}q^\star}{\mathrm{d}\mu} =\frac{e^h}{\int e^h\,\mathrm{d}\mu}. \]
Proof. Define \(q^\star\) by the displayed density. For any \(q\ll\mu\), \[ \operatorname{KL}(q\|q^\star) =\operatorname{KL}(q\|\mu)-\int h\,\mathrm{d}q +\log\int e^h\,\mathrm{d}\mu. \] Rearrangement and nonnegativity of KL prove the upper bound. Equality holds precisely when \(q=q^\star\).
This identity is simultaneously the conjugacy of negative entropy, the normalization of an exponential family, and the exact coordinate update in mean-field variational inference.
A KL-proximal step over densities
Functional mirror update. Let \(q_k\) be a probability density and let \(g\) be an integrable functional gradient. The solution of \[ q_{k+1} =\operatorname*{arg\,min}_{q} \left\{ \eta\int g\,q\,\mathrm{d}\mu +\operatorname{KL}(q\|q_k) \right\} \] over probability densities is \[ q_{k+1}(z) =\frac{q_k(z)e^{-\eta g(z)}} {\int q_k(u)e^{-\eta g(u)}\,\mathrm{d}\mu(u)}. \]
Proof. Apply the Gibbs variational principle with base measure \(q_k\,\mathrm{d}\mu\) and \(h=-\eta g\), or differentiate the Lagrangian including the normalization constraint. In either case, \(\log(q_{k+1}/q_k)=-\eta g-\text{constant}\), and normalization determines the constant.
In the finite simplex this becomes \[ q_{k+1,i} =\frac{q_{k,i}e^{-\eta g_i}} {\sum_jq_{k,j}e^{-\eta g_j}}. \] The update cannot create mass outside the support of \(q_k\); support choice is therefore part of the algorithmic model.
Coordinate variational inference as exact KL projection
For a mean-field density \(q(z)=q_j(z_j)q_{-j}(z_{-j})\), maximizing the ELBO over \(q_j\) gives \[ q_j^+(z_j) \propto \exp\{\mathbb E_{q_{-j}}\log p(x,z_j,Z_{-j})\}. \] This is a Gibbs update with a reference measure on the \(j\)-th factor. It is also an exact Bregman projection in KL geometry, rather than a small first-order step. Natural-gradient VI instead restricts \(q\) to a smooth parametric family and uses the Fisher metric on its parameters.
Entropic regularization and zero-temperature limits
Soft minimum. For real numbers \(a_1,\ldots,a_m\) and \(\tau>0\), \[ -\tau\log\sum_{i=1}^me^{-a_i/\tau} \le\min_i a_i \le-\tau\log\sum_{i=1}^me^{-a_i/\tau} +\tau\log m. \] The entropy-regularized minimizer on the simplex is \(q_i^\star\propto e^{-a_i/\tau}\).
Proof. If \(a_\star=\min_i a_i\), then \[ e^{-a_\star/\tau} \le\sum_i e^{-a_i/\tau} \le m e^{-a_\star/\tau}. \] Take \(-\tau\log\) and reverse the inequalities appropriately. The Gibbs variational principle gives the optimizer.
As \(\tau\downarrow0\), Gibbs mass concentrates on minimizers and the regularized value approaches the hard minimum. The bias is controlled by \(\tau\log m\), while positive temperature improves smoothness and numerical stability.
These same identities recur in entropic variational inference, entropic optimal transport, and entropy-regularized reinforcement learning. The objects differ, but the KL-proximal geometry is the same.