Information geometry insists that statistical constructions should not depend on the accidental coordinates chosen for a model. Reparametrization invariance is the first test of whether a formula is geometric.
Coordinate changes
A parameter transformation changes components but not the underlying distribution.
Definition. A reparametrization is a diffeomorphism \(\phi=\phi(\theta)\). Scalars, vectors, covectors, and tensors transform by the usual rules: \[ v^a=\frac{\partial\phi^a}{\partial\theta^i}v^i,\qquad \omega_a=\frac{\partial\theta^i}{\partial\phi^a}\omega_i. \]
Remark. A scalar statistical objective such as KL divergence between two fixed distributions is invariant. Its coordinate gradient is not a vector until a metric is used to raise an index.
Fisher metric as a tensor
The Fisher information matrix has the pullback transformation law of a Riemannian metric.
Proposition. Under \(\theta=\theta(\phi)\), \[ g^{(\phi)}_{ab} =\frac{\partial\theta^i}{\partial\phi^a}\frac{\partial\theta^j}{\partial\phi^b}g^{(\theta)}_{ij}. \]
Proof. The score transforms by the chain rule. Taking score inner products gives exactly the pullback formula.
Ordinary gradient is not invariant
Euclidean gradients depend on the coordinate scale.
Proposition. The coordinate gradient \(\partial_i L\) is a covector, not an invariant search direction. The natural gradient \(g^{ij}\partial_jL\) transforms as a vector.
Proof. The differential \(\mathrm{d}L\) has components \(\partial_iL\), which transform covariantly. Multiplying by the inverse metric raises the index. Because \(g^{ij}\) transforms contravariantly in two indices, \(g^{ij}\partial_jL\) has the vector transformation law.
Invariant algorithms
Natural gradient and mirror-descent formulations are attempts to make optimization respect the statistical geometry.
Remark. Invariance does not automatically guarantee good numerical behavior, but it prevents the algorithm from changing merely because the same model was written in different coordinates.
Tensorial transformation laws
Let \(\eta=\psi(\theta)\) be a smooth change of coordinates and write \(J^i{}_a=\partial\theta^i/\partial\eta^a\). Coordinate tangent vectors and score components obey \[ \partial_{\eta^a}=J^i{}_a\partial_{\theta^i}, \qquad \ell_a^{(\eta)}=J^i{}_a\ell_i^{(\theta)}. \] Consequently, \[ g_{ab}^{(\eta)} =J^i{}_aJ^j{}_b g_{ij}^{(\theta)}, \] the covariant transformation law for a \((0,2)\)-tensor.
Proposition. The line element \(g_{ij}\mathrm{d}\theta^i\mathrm{d}\theta^j\), curve length, Fisher--Rao distance, and Riemannian volume are invariant under smooth reparametrization.
Proof. Curve velocities satisfy \(\dot\theta^i=J^i{}_a\dot\eta^a\). Substitution gives \(g^{(\theta)}_{ij}\dot\theta^i\dot\theta^j =g^{(\eta)}_{ab}\dot\eta^a\dot\eta^b\), hence equal lengths and distances. Moreover, \[ \sqrt{\det g^{(\eta)}}\,\mathrm{d}\eta =|\det J|\sqrt{\det g^{(\theta)}}\,\mathrm{d}\eta =\sqrt{\det g^{(\theta)}}\,\mathrm{d}\theta. \]
Covectors, gradients, and natural motion
The differential of a scalar objective \(L\) is the covector \(\mathrm{d}L=(\partial_iL)\mathrm{d}\theta^i\). Treating its components as a vector silently uses an identity matrix as metric. Under a coordinate change, \[ \partial_a^{(\eta)}L=J^i{}_a\partial_i^{(\theta)}L, \] which is covariant, not the transformation rule for a tangent vector. Raising the index with \(g^{-1}\) repairs the mismatch: \[ (\operatorname{grad}_gL)^i=g^{ij}\partial_jL. \]
Example (Bernoulli coordinates). In probability coordinate \(p\), \(g_{pp}=1/\{p(1-p)\}\), so \[ \operatorname{grad}_gL=p(1-p)L'(p)\,\partial_p. \] In logit coordinate \(\eta=\log\{p/(1-p)\}\), \(g_{\eta\eta}=p(1-p)\) and \(\partial_\eta L=p(1-p)L'(p)\). Therefore \(g^{\eta\eta}\partial_\eta L=L'(p)\); converting \(\partial_\eta=p(1-p)\partial_p\) recovers the same vector field.
Invariant information for a functional
For a smooth scalar functional \(\psi:\mathcal M\to\mathbb R\), the intrinsic squared norm of its differential is \[ \|\mathrm{d}\psi\|_{g^{-1}}^2 =\partial_i\psi\,g^{ij}\partial_j\psi. \] This is the Cramér--Rao lower bound and is unchanged by changes of model or nuisance coordinates. A one-to-one transformation \(\varphi=a(\psi)\) changes the bound by \(\{a'(\psi)\}^2\), exactly as a variance should.
Invariance does not say that every procedure is unchanged by reparametrization. Maximum likelihood is invariant under one-to-one parameter maps; posterior modes, Euclidean penalties, and finite-step gradient algorithms generally are not unless their reference measures or geometries are transformed as well.