Information quantities are convex dual objects. Variational formulas turn entropy and relative entropy into optimization principles, useful for large deviations and exponential bounds.
Donsker-Varadhan formula
Theorem. For probability measures \(P\ll Q\) on a finite space, \[ D(P\|Q)=\sup_f\left\{\mathbb E_P f-\log \mathbb E_Q e^f\right\}, \] where the supremum ranges over all real functions \(f\). The optimizer is \(f=\log(\mathrm{d}P/\mathrm{d}Q)+c\).
Entropy duality
Corollary. The preceding identity is equivalent to \[ \log \mathbb E_Q e^f=\sup_P\{\mathbb E_P f-D(P\|Q)\}. \] Thus log-moment generating functions and relative entropy are convex conjugates.
Proof by Gibbs inequality
Proof. For any \(f\), define \(Q_f(x)=e^{f(x)}Q(x)/\mathbb E_Qe^f\). Gibbs inequality gives \(D(P\|Q_f)\ge0\), or \[ D(P\|Q)\ge \mathbb E_P f-\log\mathbb E_Qe^f. \] Equality holds when \(Q_f=P\).
Information projection
Remark. Minimizing \(D(P\|Q)\) over a convex constraint set gives an information projection. This is the geometric core behind maximum entropy distributions and Sanov conditional limit theorems.
Gibbs variational principle
Theorem. For a reference probability \(Q\) and measurable \(f\) with \(Qe^f<\infty\), \[ \log Qe^f= \sup_{P\ll Q}\{Pf-D(P\|Q)\}. \] The optimizer is \[ \frac{\mathrm{d}P_f}{\mathrm{d}Q} =\frac{e^f}{Qe^f}. \]
Proof. \[ D(P\|P_f) =D(P\|Q)-Pf+\log Qe^f\ge0. \] Rearrangement gives the upper bound; \(P=P_f\) attains equality.
Dual formula for KL divergence
Corollary. \[ D(P\|Q)= \sup_f\{Pf-\log Qe^f\}, \] where the supremum ranges over bounded measurable \(f\), or a larger integrable class when justified.
Proof. The Gibbs principle gives \(Pf-\log Qe^f\le D(P\|Q)\). For \(f=\log(\mathrm{d}P/\mathrm{d}Q)\), truncation if needed, equality is approached or attained.
Mutual information as a variational divergence
Proposition. \[ I(X;Y)= \inf_{Q_Y}\mathbb E_XD(P_{Y\mid X}\|Q_Y), \] and the unique minimizer on the relevant support is \(Q_Y=P_Y\).
Proof. For every \(Q_Y\), \[ \mathbb E_XD(P_{Y\mid X}\|Q_Y) =I(X;Y)+D(P_Y\|Q_Y) \] by adding and subtracting \(\log p_Y(Y)\). Minimize the nonnegative second term.
Donsker-Varadhan neural lower bounds
Corollary. For every critic \(f(x,y)\), \[ I(X;Y)\ge \mathbb E_{P_{XY}}f(X,Y) -\log\mathbb E_{P_XP_Y}e^{f(X,Y)}. \]
Optimizing a restricted critic class gives a lower bound, not generally the exact information. Estimating the log moment from finite samples can have high variance and bias, especially when true information is large because the optimal density ratio is heavy-tailed.
Fenchel representation of \(f\)-divergences
Theorem. Let \(D_f(P\|Q)=Q f(\mathrm{d}P/\mathrm{d}Q)\) for convex \(f\) with \(f(1)=0\). If \(f^*\) is the convex conjugate, then \[ D_f(P\|Q)\ge \sup_T\{PT-Qf^*(T)\}, \] with equality under standard regularity when the function class contains \(T=f'(\mathrm{d}P/\mathrm{d}Q)\).
Proof. Fenchel-Young gives \(ut\le f(u)+f^*(t)\). Set \(u=\mathrm{d}P/\mathrm{d}Q\), \(t=T\), integrate under \(Q\), and rearrange. Equality is the pointwise equality condition \(T\in\partial f(u)\).
Log-partition derivatives
Proposition. Let \[ A(\lambda)=\log Qe^{\lambda f}, \qquad \frac{\mathrm{d}Q_\lambda}{\mathrm{d}Q} =e^{\lambda f-A(\lambda)}. \] Then \[ A'(\lambda)=Q_\lambda f,\qquad A''(\lambda)=\operatorname{Var}_{Q_\lambda}(f)\ge0. \]
Proof. Differentiate under the integral. The first derivative divides the weighted first moment by the normalizing integral. Differentiating again gives weighted second moment minus squared weighted mean.
This convexity is the basis of Chernoff bounds, exponential families, thermodynamic integration, and entropic risk.
Entropic risk and robust expectations
Theorem. For \(\eta>0\), \[ \frac1\eta\log Qe^{\eta f} = \sup_P\left\{ Pf-\frac1\eta D(P\|Q)\right\}. \]
The left side is an entropic risk measure; the right side is a distributionally robust expectation penalized by KL distance. The optimizer exponentially tilts toward high-loss outcomes.
Variational characterization of channel capacity
Theorem. For channel \(W(\mathrm{d}y\mid x)\), \[ C=\sup_{P_X}I(X;Y) =\inf_{Q_Y}\sup_xD\{W(\cdot\mid x)\|Q_Y\}. \] This is the information-radius formula.
Proof outline. The identity \[ \mathbb E_{P_X}D(W_x\|Q) =I(P_X,W)+D(P_Y\|Q) \] gives the upper bound \(I(P_X,W)\le\sup_xD(W_x\|Q)\). Minimize over \(Q\). Convex minimax theory supplies equality for compact finite-alphabet simplices; an optimal output distribution equalizes divergence on the support of a capacity-achieving input.
Variational formulas require domains
Extended values matter. If \(P\not\ll Q\), KL is infinite and critics can diverge on a \(P\)-positive, \(Q\)-null set. Interchanging supremum, expectation, and differentiation requires measurability and integrability. A formal variational expression is not a computational guarantee until these domains and approximation errors are controlled.