\(f\)-divergences are a large family of divergences satisfying data processing. Locally, all smooth \(f\)-divergences generate Fisher information up to scale.
Convex generators
An \(f\)-divergence compares likelihood ratios through a convex function.
Definition. For convex \(f\) with \(f(1)=0\), \[ D_f(P\|Q)=\int f\left(\frac{\mathrm{d}P}{\mathrm{d}Q}\right)\,\mathrm{d}Q \] when \(P\ll Q\), with standard extensions otherwise.
Example. KL divergence corresponds to \(f(u)=u\log u\). Squared Hellinger distance corresponds, up to convention, to \(f(u)=(\sqrt u-1)^2\).
Data processing
Convexity gives monotonicity under Markov kernels.
Theorem. For any Markov kernel \(K\), \[ D_f(KP\|KQ)\le D_f(P\|Q). \]
Proof. The likelihood ratio after applying \(K\) is a conditional expectation of the original likelihood ratio under the \(Q\)-joint law. Jensen's inequality for convex \(f\) gives the pointwise conditional bound, and integrating with respect to \(KQ\) gives the result.
Local Fisher metric
All smooth \(f\)-divergences share the Fisher metric locally.
Proposition. For a regular model, \[ D_f(p_\theta\|p_{\theta+\mathrm{d}\theta}) =\frac{f''(1)}{2}g_{ij}\mathrm{d}\theta^i\mathrm{d}\theta^j+o(\|\mathrm{d}\theta\|^2). \]
Proof. The likelihood ratio between nearby distributions is \(1+s_i\mathrm{d}\theta^i+O(\|\mathrm{d}\theta\|^2)\). Taylor expand \(f\) at \(1\). The constant term vanishes and the linear term integrates to zero by normalization. The quadratic term gives \(f''(1)\mathbb E(s_is_j)\mathrm{d}\theta^i\mathrm{d}\theta^j/2\).
Why KL is still special
KL is not the only monotone divergence, but it has special additive and variational properties.
Remark. The chain rule, variational formulas, exponential-family projections, and likelihood asymptotics make KL especially central even though Fisher local geometry is shared by many divergences.
Definition and normalization
Definition. For convex \(f:(0,\infty)\to\mathbb R\) with \(f(1)=0\), \[ D_f(P\|Q)=\int q\,f\!\left(\frac pq\right)\,\mathrm{d}\mu, \] with the usual lower-semicontinuous extension on singular parts. Adding \(a(t-1)\) to \(f\) does not change \(D_f\), so one may normalize \(f'(1)=0\).
Theorem (data processing). For every Markov kernel \(K\), \[ D_f(PK\|QK)\le D_f(P\|Q). \]
Proof. Under \(Q\), let \(L=\mathrm{d}P/\mathrm{d}Q\). The output likelihood ratio is \[ \frac{\mathrm{d}(PK)}{\mathrm{d}(QK)}(Y)=E_Q(L\mid Y). \] Conditional Jensen gives \[ E_Q f\{E_Q(L\mid Y)\}\le E_Q f(L). \] The singular case follows by approximation or the perspective extension.
Equality conditions
If \(f\) is strictly convex, equality holds precisely when the likelihood ratio \(L\) is measurable with respect to the output, modulo null sets. Thus the channel is sufficient for the pair \(\{P,Q\}\). For a whole model, equality for every pair is tied to Blackwell sufficiency.
Universal local metric
Proposition. If \(f\) is \(C^3\) near \(1\), \(f(1)=f'(1)=0\), then in a regular model \[ D_f(P_{\theta+h}\|P_\theta) =\frac{f''(1)}2h^\top I_\theta h+O(\|h\|^3). \] After scaling \(f''(1)=1\), every smooth \(f\)-divergence induces the Fisher metric.
Proof. Write the likelihood ratio as \(1+r_h\), where \(r_h=h^\top\dot\ell_\theta+o_{L_2}(\|h\|)\) and \(P_\theta r_h=0\). Taylor expansion gives \[ P_\theta f(1+r_h) =\frac{f''(1)}2P_\theta r_h^2+O(P_\theta|r_h|^3), \] which yields the result under the required third-moment control.
Examples and comparisons
KL uses \(f(t)=t\log t\); reverse KL uses \(f(t)=-\log t\); total variation uses \(f(t)=|t-1|/2\); squared Hellinger uses \(f(t)=(\sqrt t-1)^2\); and Pearson chi-square uses \(f(t)=(t-1)^2\). They share data processing but differ in tail weighting, finiteness, and local smoothness. Total variation is not twice differentiable at \(1\), so it does not directly generate Fisher geometry through a Hessian.
Among \(f\)-divergences, KL is distinguished by its exact product and chain rules and its close relation to exponential families. Monotonicity alone selects the Fisher metric locally but does not uniquely select KL globally.