Divergence third derivatives explain why information geometry naturally has two affine connections. The first and second arguments of a divergence are geometrically asymmetric.
Diagonal derivative calculus
The diagonal \(p=q\) is where a contrast function becomes local geometry.
Definition. For coordinates \(\theta\) on the first argument and \(\theta'\) on the second, diagonal evaluation means setting \(\theta'=\theta\) after differentiating. Unprimed indices differentiate the first argument; primed indices differentiate the second.
Remark. Differentiating before restricting to the diagonal is essential. Restricting first would make \(D(p,p)=0\) identically and lose all geometry.
Connection coefficients
The third derivatives of a divergence define two sets of Christoffel symbols.
Proposition. Let \(D\) generate metric \(g_{ij}=-\partial_i\partial_{j'}D|_{\mathrm{diag}}\). Define \[ \Gamma_{ijk}=-\partial_i\partial_j\partial_{k'}D|_{\mathrm{diag}}, \qquad \Gamma^*_{ijk}=-\partial_{i'}\partial_{j'}\partial_kD|_{\mathrm{diag}}. \] Then \(\Gamma\) and \(\Gamma^*\) are coefficients of torsion-free dual connections.
Proof. Symmetry of second derivatives in the unprimed pair gives \(\Gamma_{ijk}=\Gamma_{jik}\), and similarly for the primed pair, so both connections are torsion-free. Differentiating the metric expression along the diagonal gives \(\partial_kg_{ij}=\Gamma_{kij}+\Gamma^*_{kji}\), which is the coordinate duality condition.
KL divergence case
For KL divergence, the induced pair is the exponential and mixture connections.
Theorem. On a regular positive model, \(D_{\mathrm{KL}}(p_\theta\|p_{\theta'})\) induces the Fisher metric and the standard dual pair of exponential and mixture connections.
Proof. The second mixed derivative of KL gives the Fisher metric by the usual score-Hessian identity. The two possible third derivatives keep track of whether differentiation acts on the source or target argument; these yield the two Amari connections \(\alpha=\pm1\) under standard sign conventions.
Choice of divergence
The divergence fixes both local metric and preferred affine structure.
Remark. A Riemannian metric alone does not determine the exponential and mixture connections. A divergence contains third-order information, and this third-order information is exactly what specifies the dual affine structure.
Eguchi's derivative construction
Let \(\mathcal D(\theta,\eta)\) be \(C^3\) near the diagonal. Adopt \[ g_{ij}=-\partial_i\partial_{j'}\mathcal D|, \quad \Gamma_{ijk}=-\partial_i\partial_j\partial_{k'}\mathcal D|, \quad \Gamma^*_{ijk}=-\partial_{i'}\partial_{j'}\partial_k\mathcal D|. \] Indices on connection coefficients are lowered with \(g\).
Theorem. If \(g\) is positive definite, the formulas define torsion-free affine connections dual with respect to \(g\).
Proof. Symmetry of repeated partial derivatives in \(i,j\) makes both connections torsion free. Differentiate \(g_{jk}=-\partial_j\partial_{k'}\mathcal D|\) along the diagonal; both arguments move, hence \[ \partial_i g_{jk} =-\partial_i\partial_j\partial_{k'}\mathcal D| -\partial_{i'}\partial_j\partial_{k'}\mathcal D| =\Gamma_{ijk}+\Gamma^*_{ikj}. \] This is exactly the coordinate form of metric duality. Raising the final index with \(g^{k\ell}\) gives affine connection coefficients.
Coordinate invariance despite non-tensorial coefficients
At first sight, third partial derivatives appear coordinate dependent. Under a coordinate change, chain-rule terms involving second derivatives of the transformation survive because first derivatives of \(\mathcal D\) vanish but its second derivatives do not. Those terms are precisely the inhomogeneous terms in the affine-connection transformation law. The construction therefore defines genuine connections, not three-tensors.
Bregman calculation
Example. For a strictly convex \(\psi\), \[ B_\psi(\theta,\eta) =\psi(\theta)-\psi(\eta)-\nabla\psi(\eta)^\top(\theta-\eta). \] The induced metric is \(g_{ij}=\partial_{ij}\psi\). In \(\theta\)-coordinates, one of the induced connections has zero coefficients, while its dual has \[ \Gamma^*_{ijk}=\partial_{ijk}\psi \] up to the argument-order convention. Thus the Hessian potential produces a dually flat geometry.
Gauge freedom
Proposition. Replacing \(\mathcal D(p,q)\) by \[ \widetilde{\mathcal D}(p,q) =\mathcal D(p,q)+a(p)+b(q) \] preserves the mixed-derivative metric and connections whenever the adjusted function still vanishes to first order on the diagonal.
Only derivatives involving both arguments enter the geometry. This nonuniqueness is another reason a divergence contains more global information than the local dualistic structure it generates.