The natural gradient is the Riemannian gradient under the Fisher metric. It turns an ordinary differential into an intrinsic vector field, correcting for parameter scale and statistical curvature.
Riemannian gradient
The gradient depends on the metric used to identify covectors and vectors.
Definition. For a smooth objective \(L\), the natural gradient \(\widetilde\nabla L\) is defined by \[ g_\theta(\widetilde\nabla L,v)=\mathrm{d}L_\theta(v) \] for all tangent vectors \(v\). In coordinates, \[ (\widetilde\nabla L)^i=g^{ij}\partial_jL. \]
Proposition. The natural gradient transforms as a vector field under reparametrization.
Proof. The differential \(\partial_jL\) is a covector and \(g^{ij}\) is the inverse metric tensor. Contracting them raises the index, so the result obeys the contravariant vector transformation law.
Steepest ascent
The natural gradient is steepest when step length is measured by Fisher distance.
Theorem. Among directions \(v\) satisfying \(g_\theta(v,v)\le\varepsilon^2\), the first-order gain \(\mathrm{d}L_\theta(v)\) is maximized by \(v\) proportional to \(\widetilde\nabla L\).
Proof. By Cauchy-Schwarz in the Fisher inner product, \[ \mathrm{d}L(v)=g(\widetilde\nabla L,v)\le \|\widetilde\nabla L\|_g\|v\|_g. \] Equality holds when \(v\) is a positive multiple of \(\widetilde\nabla L\).
KL trust region
Natural gradient is the infinitesimal solution of a KL-constrained optimization problem.
Proposition. For small steps, maximizing \(L(\theta+\Delta)\) subject to \(D_{\mathrm{KL}}(p_\theta\|p_{\theta+\Delta})\le\varepsilon\) gives \(\Delta\propto g^{-1}\nabla L\) to first order.
Proof. Use the expansions \(L(\theta+\Delta)=L(\theta)+\partial_iL\Delta^i+o(\|\Delta\|)\) and \(D_{\mathrm{KL}}=\frac12g_{ij}\Delta^i\Delta^j+o(\|\Delta\|^2)\). The Lagrange multiplier equation for the quadratic constraint gives \(\partial_iL=\lambda g_{ij}\Delta^j\).
Relation with mirror descent
In dually flat coordinates, natural-gradient flow and mirror descent are two descriptions of the same geometry.
Remark. When Fisher geometry is Hessian, the metric potential defines a mirror map. Natural-gradient steps in primal coordinates correspond to ordinary gradient-like steps in dual coordinates up to discretization choices.
Metric steepest descent
Let \(L:\mathcal M\to\mathbb R\). The Riemannian gradient is the unique vector satisfying \[ g_\theta(\operatorname{grad}_gL,v)=\mathrm{d}L_\theta(v) \quad\text{for every }v. \] In coordinates, \(\operatorname{grad}_gL=I(\theta)^{-1}\nabla_\theta L\). For minimization, the natural-gradient flow is \(\dot\theta=-I(\theta)^{-1}\nabla L(\theta)\).
Theorem (intrinsic steepest direction). For \(\varepsilon>0\), \[ \min_{v:g_\theta(v,v)\le\varepsilon^2}\mathrm{d}L_\theta(v) =-\varepsilon\|\operatorname{grad}_gL\|_g, \] attained at \[ v=-\varepsilon\, \frac{\operatorname{grad}_gL}{\|\operatorname{grad}_gL\|_g} \] when the gradient is nonzero.
Proof. Metric Cauchy--Schwarz gives \[ \mathrm{d}L(v)=g(\operatorname{grad}_gL,v) \ge-\|\operatorname{grad}_gL\|_g\|v\|_g. \] The displayed vector attains equality.
KL trust regions
Proposition. The small-step problem \[ \min_\delta \nabla L(\theta)^\top\delta \quad\text{subject to}\quad D(P_\theta\|P_{\theta+\delta})\le\rho \] has leading solution \[ \delta =-\sqrt{\frac{2\rho} {\nabla L^\top I^{-1}\nabla L}}\, I^{-1}\nabla L. \]
Proof. Replace KL by \(\tfrac12\delta^\top I\delta+o(\|\delta\|^2)\). The Lagrange equation is \(\nabla L+\lambda I\delta=0\); imposing the active quadratic constraint determines \(\lambda\).
Relation to mirror descent
If \(I(\theta)=\nabla^2\psi(\theta)\), mirror descent with step \(\alpha\) updates dual coordinates by \[ \eta_{t+1}=\eta_t-\alpha\nabla L(\theta_t), \qquad \eta_t=\nabla\psi(\theta_t). \] Taylor expansion of \(\nabla\psi^{-1}\) gives \[ \theta_{t+1} =\theta_t-\alpha I(\theta_t)^{-1}\nabla L(\theta_t)+O(\alpha^2). \] Thus the flows agree infinitesimally, while finite steps differ because mirror descent follows dual-affine geometry and a naive natural-gradient Euler step uses coordinate addition.
Singular and stochastic information matrices
In overparameterized models, \(I\) may be singular. The Moore--Penrose direction \(I^\dagger\nabla L\) is intrinsic only after restricting the differential to identifiable tangent directions; damping \((I+\lambda I_d)^{-1}\) introduces coordinate-dependent Euclidean geometry. For stochastic gradients, estimates of Fisher or its block/diagonal approximations trade invariance for computation. The empirical outer product of sample gradients equals Fisher only under correct likelihood specification and the appropriate expectation.