Legendre duality turns one affine coordinate system into its dual. In information geometry it relates natural parameters, expectation parameters, and entropy-like potentials.
Legendre transform
A strictly convex potential defines a dual potential by optimizing a linear perturbation.
Definition. For a strictly convex differentiable \(\psi(\theta)\), \[ \psi^*(\eta)=\sup_\theta\{\theta^i\eta_i-\psi(\theta)\}. \] When \(\eta_i=\partial_i\psi(\theta)\), the supremum is attained at \(\theta\).
Proposition. The gradients are inverse maps: \[ \eta=\nabla\psi(\theta),\qquad \theta=\nabla\psi^*(\eta). \]
Proof. The first-order condition for the supremum is \(\eta=\nabla\psi(\theta)\). Differentiating the identity \(\psi^*(\eta)=\theta(\eta)\cdot\eta-\psi(\theta(\eta))\) gives \(\nabla\psi^*(\eta)=\theta(\eta)\), because the terms involving \(\mathrm{d}\theta/\mathrm{d}\eta\) cancel by the first-order condition.
Dual Hessians
The two Hessian metrics are matrix inverses.
Theorem. If \(g_{ij}=\partial_i\partial_j\psi\) and \(g^{ij}_*=\partial^i\partial^j\psi^*\), then the Hessian matrices are inverse under the coordinate transformation \(\eta=\nabla\psi(\theta)\).
Proof. Differentiate \(\eta_i=\partial_i\psi(\theta)\) to get \(\mathrm{d}\eta=g\,\mathrm{d}\theta\). Differentiate \(\theta=\nabla\psi^*(\eta)\) to get \(\mathrm{d}\theta=g_*\,\mathrm{d}\eta\). Composing the two differentials yields \(g_*g=I\).
Fenchel equality
Dual potentials satisfy an exact equality at matching coordinates.
Proposition. If \(\eta=\nabla\psi(\theta)\), then \[ \psi(\theta)+\psi^*(\eta)=\theta^i\eta_i. \]
Proof. At the maximizing \(\theta\), the definition of \(\psi^*\) gives \(\psi^*(\eta)=\theta^i\eta_i-\psi(\theta)\).
Statistical meaning
In exponential families, the log-partition function and negative entropy are Legendre dual.
Remark. Natural parameters are \(\theta\), expectation parameters are \(\eta=\mathbb E_\theta T(X)\), the cumulant function is \(\psi\), and the dual potential is a negative entropy function on the mean-parameter space.
Legendre-type convex functions
Definition. A proper closed convex function \(\psi\) is of Legendre type when it is strictly convex on the interior of its domain and essentially smooth: it is differentiable there and \(\|\nabla\psi(\theta_n)\|\to\infty\) whenever \(\theta_n\) approaches a finite boundary point. Its convex conjugate is \[ \psi^*(\eta)=\sup_\theta\{\theta^\top\eta-\psi(\theta)\}. \]
Theorem (Legendre reciprocity). If \(\psi\) is Legendre type, then \[ \nabla\psi:\operatorname{int}\operatorname{dom}\psi \longrightarrow\operatorname{int}\operatorname{dom}\psi^* \] is one-to-one with inverse \(\nabla\psi^*\). At paired points \(\eta=\nabla\psi(\theta)\), \[ \psi(\theta)+\psi^*(\eta)=\theta^\top\eta. \]
Proof. Fenchel--Young gives \(\psi(\theta)+\psi^*(\eta)\ge\theta^\top\eta\), with equality exactly when \(\eta\in\partial\psi(\theta)\). Differentiability and strict convexity make the subgradient single-valued and injective. Essential smoothness supplies the onto statement for interior dual domains. Equality then gives the inverse gradient relation.
Hessian reciprocity
Proposition. If \(\psi\in C^2\) and \(\nabla^2\psi\succ0\), then at dual points, \[ \nabla^2\psi^*(\eta) =\{\nabla^2\psi(\theta)\}^{-1}. \]
Proof. Differentiate \(\nabla\psi^*(\nabla\psi(\theta))=\theta\) and apply the chain rule.
Log-partition and negative entropy
For a minimal exponential family, the log-partition function \[ \psi(\theta)=\log\int e^{\theta^\top T(x)}h(x)\,\mathrm{d}\mu(x) \] has \[ \nabla\psi(\theta)=E_\theta T,\qquad \nabla^2\psi(\theta)=\operatorname{Cov}_\theta(T). \] Its conjugate at a realizable mean \(\eta\) equals negative maximum entropy relative to \(h\): \[ \psi^*(\eta) =\sup_\theta\{\theta^\top\eta-\psi(\theta)\} =-H(P_{\theta(\eta)})-E_{\theta(\eta)}\log h(X). \] Thus natural and mean parameters are Legendre dual, and Fisher information is the Hessian linking them.
Example (categorical family). Taking one category as reference, \[ \psi(\theta)=\log\!\left(1+\sum_{i=1}^{m-1}e^{\theta_i}\right), \qquad \eta_i=\frac{e^{\theta_i}}{1+\sum_j e^{\theta_j}}. \] The conjugate is \(\psi^*(p)=\sum_{i=1}^m p_i\log p_i\), negative Shannon entropy. The Hessian \(\operatorname{diag}(p)-pp^\top\) is the covariance of categorical sufficient statistics.