A differential is coordinate-independent, but its Euclidean gradient is not: identifying covectors with vectors requires a metric. Natural gradient uses the Fisher information metric, so the update reflects local statistical distinguishability rather than arbitrary parameter units.
Raising the differential with a metric
Definition. Let \(G(\theta)\) be a positive-definite Riemannian metric matrix. The Riemannian gradient \(\operatorname{grad}_G L(\theta)\) is defined by \[ v^\top G(\theta)\operatorname{grad}_G L(\theta) =\mathrm{d}L_\theta(v) =v^\top\nabla_\theta L(\theta) \quad\text{for every }v. \] Hence \[ \operatorname{grad}_G L(\theta) =G(\theta)^{-1}\nabla_\theta L(\theta). \]
The ordinary gradient is the special case \(G=I\). The inverse metric does not merely rescale a vector after optimization; it is the mathematical map that turns the differential into a tangent direction.
Reparametrization covariance
Coordinate invariance. Let \(\theta=\phi(\eta)\) be a smooth invertible reparametrization with Jacobian \(J=D\phi(\eta)\). If \[ G_\eta(\eta)=J^\top G_\theta(\theta)J, \] then the two coordinate representations of the Riemannian gradient satisfy \[ J\,\operatorname{grad}_{G_\eta}L(\eta) =\operatorname{grad}_{G_\theta}L(\theta). \]
Proof. The chain rule gives \(\nabla_\eta L=J^\top\nabla_\theta L\). Therefore \[ JG_\eta^{-1}\nabla_\eta L =J(J^\top G_\theta J)^{-1}J^\top\nabla_\theta L =G_\theta^{-1}\nabla_\theta L, \] using invertibility of \(J\). This is the desired transformation law.
Fisher information from local KL divergence
For a regular statistical model \(p_\theta\), the Fisher information is \[ I(\theta) =\mathbb E_\theta[ \nabla_\theta\log p_\theta(X) \nabla_\theta\log p_\theta(X)^\top]. \] A second-order expansion gives \[ \operatorname{KL}(p_\theta\|p_{\theta+\delta}) =\frac12\delta^\top I(\theta)\delta+o(\|\delta\|^2). \] The linear term vanishes because the score has mean zero. Thus the Fisher quadratic form is the local distinguishability cost of changing the model.
The KL trust-region calculation
Natural steepest-ascent direction. The solution of the local problem \[ \max_\delta\ \nabla L(\theta)^\top\delta \quad\text{subject to}\quad \frac12\delta^\top I(\theta)\delta\le\varepsilon \] is \[ \delta^\star = \sqrt{\frac{2\varepsilon} {\nabla L^\top I^{-1}\nabla L}}\, I(\theta)^{-1}\nabla L(\theta). \]
Proof. The constraint is active unless \(\nabla L=0\). The Lagrangian stationarity equation is \(\nabla L-\lambda I\delta=0\), so \(\delta=\lambda^{-1}I^{-1}\nabla L\). Substitute into the quadratic constraint to determine \(\lambda\) and obtain the formula.
Natural-gradient ascent uses a chosen step length in this direction: \[ \theta_{k+1} =\theta_k+\eta_kI(\theta_k)^{-1}\nabla L(\theta_k). \] When information is singular, the model is locally nonidentifiable; damping or a pseudoinverse is then needed, and invariance must be interpreted on the identifiable tangent space.
Exponential-family dual coordinates
Example. In an exponential family \(p_\theta(x)=\exp\{\theta^\top T(x)-A(\theta)\}h(x)\), \[ I(\theta)=\nabla^2A(\theta),\qquad \mu=\nabla A(\theta)=\mathbb E_\theta T(X). \] Since \(\mathrm{d}\mu=I(\theta)\,\mathrm{d}\theta\), a natural-gradient step in \(\theta\) is an ordinary gradient step in the expectation coordinate \(\mu\), to first order. The Legendre pair \(A,A^*\) makes this the same dual-coordinate mechanism as mirror descent.
The intrinsic geometry and its regularity assumptions are developed in natural gradient and the Fisher information metric. This page supplies the optimization derivation.