Regular exponential families are the central dually flat statistical manifolds. Natural and expectation parameters are dual affine coordinates, and KL divergence is the canonical Bregman divergence.
Natural form
The log-partition function is the convex potential.
Definition. A regular exponential family has densities \[ p_\theta(x)=\exp\{\theta^iT_i(x)-\psi(\theta)+c(x)\}. \] The natural parameter is \(\theta\), and \(\psi(\theta)=\log\int e^{\theta^iT_i(x)+c(x)}\,\mathrm{d}\mu(x)\).
Proposition. The expectation parameter is \[ \eta_i=\mathbb E_\theta T_i=\partial_i\psi(\theta). \]
Proof. Differentiate the normalizing identity \(1=\int e^{\theta^iT_i-\psi(\theta)+c}\,\mathrm{d}\mu\). The derivative gives \(0=\mathbb E_\theta T_i-\partial_i\psi(\theta)\).
Fisher metric as Hessian
The cumulant potential generates the Fisher metric.
Theorem. For a regular exponential family, \[ g_{ij}=\partial_i\partial_j\psi(\theta)=\operatorname{Cov}_\theta(T_i,T_j). \]
Proof. The score is \(s_i=T_i-\partial_i\psi\). Its covariance is \(\operatorname{Cov}(T_i,T_j)\). Differentiating \(\eta_i=\partial_i\psi=\mathbb E T_i\) gives \(\partial_j\eta_i=\operatorname{Cov}(T_i,T_j)\), hence the Hessian identity.
KL as Bregman divergence
KL between two members is the Bregman divergence of \(\psi\).
Proposition. For \(p_{\theta_1},p_{\theta_2}\), \[ D(p_{\theta_1}\|p_{\theta_2}) =\psi(\theta_2)-\psi(\theta_1)-(\theta_2^i-\theta_1^i)\eta_i(\theta_1). \]
Proof. Take the expectation under \(p_{\theta_1}\) of \(\log p_{\theta_1}-\log p_{\theta_2}\). Terms involving \(c(x)\) cancel, and \(\mathbb E_{\theta_1}T_i=\eta_i(\theta_1)\).
Flatness
Natural parameters are exponential-affine; expectation parameters are mixture-affine.
Remark. This dual flatness is why exponential families are the canonical playground for information projections and maximum entropy.
Regularity and minimality
Consider \[ p_\theta(x) =\exp\{\theta^\top T(x)-\psi(\theta)\}h(x), \qquad \Theta=\left\{\theta:\int e^{\theta^\top T}h\,\mathrm{d}\mu<\infty\right\}^{\circ}. \] The family is minimal when no nonzero \(a\) makes \(a^\top T\) constant almost surely. Minimality is equivalent to strict positive definiteness of \(\nabla^2\psi\) and to local identifiability of natural parameters.
Theorem. On a minimal regular exponential family, \[ \eta(\theta)=\nabla\psi(\theta)=E_\theta T, \qquad g(\theta)=\nabla^2\psi(\theta)=\operatorname{Cov}_\theta(T), \] natural coordinates are exponential-affine, and mean coordinates are mixture-affine. The family is dually flat.
Proof. Differentiating the log normalizer gives the mean and covariance identities. Since \(\partial_{ij}\log p_\theta=-\partial_{ij}\psi\) is independent of \(x\), the exponential-connection coefficients vanish in \(\theta\). The Hessian metric and Legendre transform imply that \(\eta=\nabla\psi\) is a coordinate system with vanishing dual coefficients.
KL as the canonical divergence
Proposition. \[ D(P_\theta\|P_\xi) =\psi(\xi)-\psi(\theta)-(\xi-\theta)^\top\eta(\theta) =B_\psi(\xi,\theta) =B_{\psi^*}(\eta(\theta),\eta(\xi)). \]
The orientation matters. A straight line in natural coordinates and a straight line in mean coordinates are dual geodesics, while KL is the canonical Bregman divergence linking their endpoints.
Maximum likelihood as geometric projection
For observations \(x_1,\ldots,x_n\), the log likelihood is \[ \ell_n(\theta) =n\{\theta^\top\bar T-\psi(\theta)\}+\text{constant}. \] An interior MLE solves \(\eta(\widehat\theta)=\bar T\), so the empirical sufficient statistic is projected into the mean-parameter geometry. Its observed negative Hessian is \(n\nabla^2\psi(\widehat\theta)\), and asymptotic covariance is \(g(\theta_0)^{-1}/n\).
Proposition (boundary criterion). A finite MLE exists exactly when \(\bar T\) lies in the interior of the convex support of \(T\), under the usual full-family regularity. If \(\bar T\) lies on a face, maximizing sequences send natural parameters toward infinity and converge to an extended-family distribution on that face.
Curved exponential families
A smooth restriction \(\theta=\theta(u)\), \(u\in\mathbb R^k\), is a curved exponential family. It inherits Fisher metric by pullback but usually loses exponential flatness. The component of ambient acceleration normal to the submanifold is statistical curvature; it controls second-order information loss relative to the full family and appears in higher-order asymptotic efficiency.