KL divergence is the central example because it is both statistically operational and geometrically generative. Its second and third derivatives recover Fisher geometry and dual connections.
Definition and asymmetry
KL divergence is directed: its two arguments have different statistical roles.
Definition. For \(P\ll Q\), \[ D_{\mathrm{KL}}(P\|Q)=\int \log\frac{\mathrm{d}P}{\mathrm{d}Q}\,\mathrm{d}P. \] In densities, \(D_{\mathrm{KL}}(p\|q)=\int p\log(p/q)\,\mathrm{d}\mu\).
Proposition. KL divergence is nonnegative and equals zero iff \(P=Q\) almost surely.
Proof. Let \(Z=\mathrm{d}Q/\mathrm{d}P\). Since \(-\log\) is convex, \[ D_{\mathrm{KL}}(P\|Q)=\mathbb E_P[-\log Z]\ge -\log\mathbb E_PZ=0. \] Equality in Jensen requires \(Z\) to be constant \(P\)-a.s.; normalization gives \(Z=1\).
Fisher metric from KL
The local quadratic term of KL is Fisher information.
Theorem. For a regular model, \[ D_{\mathrm{KL}}(p_\theta\|p_{\theta+\mathrm{d}\theta}) =\frac12g_{ij}\mathrm{d}\theta^i\mathrm{d}\theta^j+o(\|\mathrm{d}\theta\|^2). \]
Proof. Taylor expand \(\ell_{\theta+\mathrm{d}\theta}\) inside \(\mathbb E_\theta(\ell_\theta-\ell_{\theta+\mathrm{d}\theta})\). The linear score term has mean zero. The expected Hessian term is \(-g_{ij}\), producing the factor \(1/2\).
Chain rule
KL has an additive decomposition under joint distributions.
Proposition. For joint laws \(P_{XY},Q_{XY}\), \[ D(P_{XY}\|Q_{XY})=D(P_X\|Q_X)+\mathbb E_{P_X}D(P_{Y|X}\|Q_{Y|X}). \]
Proof. Factor the likelihood ratio as \[ \frac{\mathrm{d}P_{XY}}{\mathrm{d}Q_{XY}} =\frac{\mathrm{d}P_X}{\mathrm{d}Q_X}\frac{\mathrm{d}P_{Y|X}}{\mathrm{d}Q_{Y|X}}. \] Taking \(P\)-expectations of the logarithm gives the two terms.
Projection behavior
KL's asymmetry leads to different forward and reverse projection geometries.
Remark. Forward KL tends to cover mass under the first distribution, while reverse KL can be mode-seeking. In dually flat geometry these are tied to mixture and exponential projections.
Differential structure of relative entropy
For dominated laws, \[ D(P\|Q)=\int p\log\frac pq\,\mathrm{d}\mu. \] It is nonnegative by Jensen's inequality, equals zero exactly when \(P=Q\), and is invariant under changes of dominating measure. Its asymmetry is statistical: the first argument supplies the averaging law.
Theorem. For a regular model, \(D(P_\theta\|P_\eta)\) induces the Fisher metric and the exponential--mixture dual pair.
Proof. Differentiate \[ D(\theta,\eta) =P_\theta\{\ell_\theta-\ell_\eta\}. \] At \(\eta=\theta\), first derivatives vanish. A mixed derivative gives \[ -\partial_i\partial_{j'}D| =P_\theta(\partial_i\ell_\theta\,\partial_j\ell_\theta) =g_{ij}. \] Third mixed derivatives, using the score-centering identities, yield the two connections with coefficients \[ \Gamma^{(e)}_{ijk}=P(\partial_{ij}\ell\,\partial_k\ell), \qquad \Gamma^{(m)}_{ijk} =P\{(\partial_{ij}\ell+\partial_i\ell\,\partial_j\ell) \partial_k\ell\}, \] up to the stated argument convention. Their difference is \(C_{ijk}\).
Chain rule and conditional geometry
Proposition (chain rule). For joint laws \(P_{XY}\) and \(Q_{XY}\), \[ D(P_{XY}\|Q_{XY}) =D(P_X\|Q_X) +E_{P_X}D(P_{Y\mid X}\|Q_{Y\mid X}). \]
Proof. Factor the likelihood ratio: \[ \log\frac{p_{XY}}{q_{XY}} =\log\frac{p_X}{q_X} +\log\frac{p_{Y\mid X}}{q_{Y\mid X}}, \] then integrate under \(P_{XY}\). Nonnegativity of the conditional term gives data processing under marginalization.
Exponential-family form
For \(p_\theta=\exp\{\theta^\top T-\psi(\theta)\}h\), \[ D(P_\theta\|P_\eta) =\psi(\eta)-\psi(\theta)-(\eta-\theta)^\top\nabla\psi(\theta) =B_\psi(\eta,\theta). \] The argument reversal is important: KL is a Bregman divergence of the log-partition function with reversed parameter order. In mean coordinates it is the Bregman divergence generated by the convex conjugate \(\psi^*\) in the opposite dual orientation.
KL is canonical, not metric
Relative entropy is additive for products, obeys a chain rule, contracts under Markov kernels, and generates the canonical dual pair. It nevertheless fails symmetry and the triangle inequality. Symmetrized KL, \[ J(P,Q)=D(P\|Q)+D(Q\|P), \] retains the same local metric but generally still fails the triangle inequality. Fisher--Rao distance and KL divergence answer different geometric questions.