The canonical divergence of a dually flat manifold is the Bregman divergence determined by its potential. It is the global contrast compatible with the local dualistic geometry.
Definition in dual coordinates
The divergence pairs a point in primal coordinates with a point in dual coordinates.
Definition. For points \(p,q\) with primal coordinates \(\theta(p),\theta(q)\) and dual coordinates \(\eta(p),\eta(q)\), \[ D(p,q)=\psi(\theta(p))+\psi^*(\eta(q))-\theta^i(p)\eta_i(q). \]
Proposition. The canonical divergence is nonnegative and vanishes exactly on the diagonal.
Proof. By Fenchel's inequality, \(\psi(\theta)+\psi^*(\eta)\ge\theta^i\eta_i\). Equality holds exactly when \(\eta=\nabla\psi(\theta)\), which means the two coordinate descriptions refer to the same point.
Bregman form
In primal coordinates, the canonical divergence is a Bregman divergence.
Proposition. The canonical divergence can be written as \[ D(p,q)=\psi(\theta_p)-\psi(\theta_q)-(\theta_p^i-\theta_q^i)\partial_i\psi(\theta_q). \]
Proof. Use \(\psi^*(\eta_q)=\theta_q^i\eta_i(q)-\psi(\theta_q)\) and \(\eta_i(q)=\partial_i\psi(\theta_q)\) in the definition.
Metric and connections
The canonical divergence regenerates the Hessian metric and the dual flat connections.
Theorem. The second mixed derivatives of \(D\) give \(g\), and the third derivatives give the flat dual connections associated with \(\theta\) and \(\eta\).
Proof. In Bregman form, differentiating twice at the diagonal gives the Hessian \(\partial_i\partial_j\psi\). Third derivatives with respect to the first affine coordinate vanish for the flat primal connection, while the corresponding dual derivatives encode the dual affine structure through the Legendre transform.
Asymmetry
The canonical divergence is generally asymmetric because the two affine coordinate systems play different roles.
Remark. Symmetry occurs essentially in the quadratic Euclidean case. In statistical examples, asymmetry is a feature: it records the difference between exponential and mixture linearity.
Definition from a dual affine structure
On a dually flat manifold with paired coordinates \(\eta=\nabla\psi(\theta)\) and \(\varphi=\psi^*\), define \[ \mathcal D(p,q) =\psi(\theta(p))+\varphi(\eta(q)) -\theta(p)^\top\eta(q). \] Fenchel--Young immediately gives \(\mathcal D(p,q)\ge0\), with equality only at \(p=q\) under strict convexity.
Proposition. \[ \mathcal D(p,q) =B_\psi(\theta(p),\theta(q)) =B_\varphi(\eta(q),\eta(p)). \]
Proof. At \(q\), \(\varphi(\eta(q))=\theta(q)^\top\eta(q)-\psi(\theta(q))\). Substitute this into the definition: \[ \mathcal D(p,q) =\psi(\theta(p))-\psi(\theta(q)) -\eta(q)^\top\{\theta(p)-\theta(q)\}. \] This is \(B_\psi\). Legendre reciprocity gives the dual form.
Recovery of metric and connections
Theorem. Diagonal derivatives of \(\mathcal D\) recover the Hessian metric and the primal--dual flat connections: \[ -\partial_i\partial_{j'}\mathcal D|=g_{ij}, \] while the appropriate third mixed derivatives vanish in primal affine coordinates for \(\nabla\) and in dual affine coordinates for \(\nabla^*\).
Therefore canonical divergence is not an arbitrary loss placed on the manifold: it integrates the complete dually flat structure. Conversely, the same metric alone cannot determine it because third-order duality data are missing.
Symmetrization and energy
Proposition. The symmetrized canonical divergence satisfies \[ \mathcal D(p,q)+\mathcal D(q,p) =\{\theta(p)-\theta(q)\}^\top\{\eta(p)-\eta(q)\}. \]
This is nonnegative by monotonicity of \(\nabla\psi\). For the straight primal segment \(\theta(t)=\theta(q)+t\{\theta(p)-\theta(q)\}\), \[ \mathcal D(p,q)+\mathcal D(q,p) =\int_0^1 g_{\theta(t)}(\dot\theta,\dot\theta)\,\mathrm{d}t. \] Thus symmetrized divergence equals the energy of a particular affine path, not generally the squared Fisher--Rao distance, which minimizes energy over all paths.
Examples
For Euclidean potential \(\psi(\theta)=\|\theta\|^2/2\), \(\mathcal D(p,q)=\|\theta(p)-\theta(q)\|^2/2\). For an exponential family, it is KL with an orientation determined by which point supplies natural and mean coordinates. For positive definite matrices with \(\psi(\Theta)=-\log\det\Theta\), it becomes the LogDet divergence \[ \operatorname{tr}(\Theta_q^{-1}\Theta_p) -\log\det(\Theta_q^{-1}\Theta_p)-d. \]