The Fisher metric is the second-order part of several statistical divergences. It therefore quantifies local distinguishability of nearby distributions.
Second-order KL expansion
Nearby models are separated quadratically by Fisher information.
Theorem. For a regular model, \[ D_{\mathrm{KL}}(p_\theta\|p_{\theta+\mathrm{d}\theta}) =\frac12 g_{ij}(\theta)\,\mathrm{d}\theta^i\mathrm{d}\theta^j+o(\|\mathrm{d}\theta\|^2). \]
Proof. Expand \(\ell_{\theta+\mathrm{d}\theta}\) around \(\theta\): \[ \ell_{\theta+\mathrm{d}\theta}=\ell_\theta+s_i\mathrm{d}\theta^i+\frac12\partial_i\partial_j\ell_\theta\,\mathrm{d}\theta^i\mathrm{d}\theta^j+o(\|\mathrm{d}\theta\|^2). \] Taking \(\mathbb E_\theta(\ell_\theta-\ell_{\theta+\mathrm{d}\theta})\), the first-order term vanishes because \(\mathbb E_\theta s_i=0\). The expected Hessian identity gives the quadratic term.
Hellinger and chi-square geometry
Different divergences share the same local metric up to scale.
Proposition. The squared Hellinger distance satisfies \[ H^2(p_\theta,p_{\theta+\mathrm{d}\theta}) =\frac18 g_{ij}(\theta)\,\mathrm{d}\theta^i\mathrm{d}\theta^j+o(\|\mathrm{d}\theta\|^2) \] under the convention \(H^2(p,q)=\frac12\int(\sqrt p-\sqrt q)^2\,\mathrm{d}\mu\).
Proof. Differentiate \(\sqrt{p_\theta}\): \(\partial_i\sqrt p=\frac12s_i\sqrt p\). The squared \(L^2(\mu)\) norm of the first-order change is \(\frac14 g_{ij}\mathrm{d}\theta^i\mathrm{d}\theta^j\). The extra factor \(1/2\) in the Hellinger convention gives \(1/8\).
Cramer-Rao interpretation
The same metric controls local estimation lower bounds.
Theorem. For an unbiased estimator \(T\) of a scalar parameter \(\psi(\theta)\), \[ \operatorname{Var}_\theta(T)\ge \partial_i\psi\,g^{ij}\,\partial_j\psi. \]
Proof. Differentiate \(\mathbb E_\theta T=\psi(\theta)\) to obtain \(\partial_i\psi=\operatorname{Cov}_\theta(T,s_i)\). For any vector \(a\), \(a^i\partial_i\psi=\operatorname{Cov}(T,a^is_i)\). Cauchy-Schwarz gives \((a^i\partial_i\psi)^2\le\operatorname{Var}(T)a^ig_{ij}a^j\). Optimize over \(a\) to obtain the bound.
Local experiments
The Fisher metric is the quadratic geometry behind LAN and Gaussian local approximations.
Remark. For \(n\) i.i.d. observations, Fisher information scales by \(n\), so local alternatives of size \(n^{-1/2}\) have finite statistical separation. This is why Fisher geometry appears throughout asymptotic statistics.
Second-order separation of nearby laws
Theorem (local divergence expansion). For a quadratic-mean differentiable model at \(\theta\), \[ D_{\mathrm{KL}}(P_\theta\|P_{\theta+h}) =\frac12h^\top I_\theta h+o(\|h\|^2), \] and, for squared Hellinger distance \(H^2(P,Q)=\int(\sqrt p-\sqrt q)^2\,\mathrm{d}\mu\), \[ H^2(P_\theta,P_{\theta+h}) =\frac14h^\top I_\theta h+o(\|h\|^2). \]
Proof. The Hellinger statement is the squared norm in the defining quadratic-mean expansion. Under the additional uniform-integrability conditions needed to expand the log likelihood, \[ \log\frac{p_{\theta+h}}{p_\theta} =h^\top\dot\ell_\theta -\frac12h^\top I_\theta h+r_h, \qquad P_\theta|r_h|=o(\|h\|^2). \] Taking \(-P_\theta\) and using \(P_\theta\dot\ell_\theta=0\) gives the KL expansion. Both therefore induce the same quadratic form up to normalization.
More generally, if an \(f\)-divergence is normalized by \(f(1)=f'(1)=0\) and \(f''(1)>0\), then \[ D_f(P_{\theta+h},P_\theta) =\frac{f''(1)}2h^\top I_\theta h+o(\|h\|^2). \] Fisher information is consequently the universal local metric of a broad class of information losses, even though their global geometries differ.
Local asymptotic normality
Theorem (LAN consequence). For iid data from a regular model and fixed \(h\), \[ \log\frac{\mathrm{d}P_{\theta+h/\sqrt n}^{\,n}} {\mathrm{d}P_\theta^{\,n}} =h^\top\Delta_{n,\theta} -\frac12h^\top I_\theta h+o_{P_\theta}(1), \qquad \Delta_{n,\theta} =\frac1{\sqrt n}\sum_{i=1}^n\dot\ell_\theta(X_i) \rightsquigarrow N(0,I_\theta). \]
The local experiment is therefore Gaussian shift: \(\Delta\sim N(I_\theta h,I_\theta)\). In that limit, \(\|h\|_{I_\theta}=(h^\top I_\theta h)^{1/2}\) controls testing power, estimation precision, and contiguity. This is the operational content of the Fisher line element.
Example (local testing power). For a scalar regular model, the score test of \(H_0:\theta=\theta_0\) rejects when \(\Delta_n/\sqrt{I_{\theta_0}}>z_{1-\alpha}\). Under \(\theta_n=\theta_0+h/\sqrt n\), Le Cam's third lemma gives limiting power \[ 1-\Phi\!\left(z_{1-\alpha}-h\sqrt{I_{\theta_0}}\right). \] Two directions having equal Fisher length have equal first-order distinguishability.
Information bounds as metric duality
Theorem (Cramér--Rao). If \(T\) is unbiased for a differentiable scalar functional \(\psi(\theta)\), then \[ \operatorname{Var}_\theta(T) \ge \nabla\psi(\theta)^\top I_\theta^{-1}\nabla\psi(\theta). \]
Proof. Differentiating \(E_\theta T=\psi(\theta)\) gives \(\operatorname{Cov}_\theta(T,\dot\ell_\theta)=\nabla\psi(\theta)\). For any \(a\), \[ \{a^\top\nabla\psi\}^2 \le \operatorname{Var}(T)\,a^\top I_\theta a. \] Choose \(a=I_\theta^{-1}\nabla\psi\). Equivalently, the differential \(\mathrm{d}\psi\) is a cotangent vector and the right side is its squared norm under the inverse Fisher metric.