The generalized Pythagorean theorem is the geometric heart of information projection. It combines one primal geodesic, one dual geodesic, and Fisher orthogonality.
Dual geodesics
In a dually flat manifold, primal and dual geodesics are straight in different coordinate systems.
Definition. A \(\nabla\)-geodesic is affine in \(\theta\)-coordinates. A \(\nabla^*\)-geodesic is affine in \(\eta\)-coordinates. These two notions of straightness are dual through \(g\).
Remark. For exponential families, exponential geodesics are straight in natural parameters; mixture geodesics are straight in expectation or probability coordinates.
Three-point identity
The Pythagorean theorem follows from a Bregman three-point identity.
Lemma. For the canonical divergence, \[ D(p,r)-D(p,q)-D(q,r) =\langle \theta(q)-\theta(p),\,\eta(r)-\eta(q)\rangle. \]
Proof. Write each divergence in Bregman form and cancel the potential terms. The remaining linear terms collect into the displayed pairing between primal and dual coordinate differences.
Generalized Pythagorean theorem
Orthogonality of primal and dual geodesics makes the three-point term vanish.
Theorem. If the \(\nabla\)-geodesic from \(q\) to \(p\) and the \(\nabla^*\)-geodesic from \(q\) to \(r\) meet \(g\)-orthogonally at \(q\), then \[ D(p,r)=D(p,q)+D(q,r). \]
Proof. The tangent of the primal geodesic is represented by \(\theta(p)-\theta(q)\); the tangent of the dual geodesic is represented by \(\eta(r)-\eta(q)\). Orthogonality makes their pairing zero. The three-point identity then gives the equality.
Projection interpretation
Information projection is a Pythagorean footpoint in dual geometry.
Remark. Many maximum-likelihood, maximum-entropy, and variational projection statements are concrete versions of this theorem.
Primal and dual affine segments
In dual affine coordinates, the \(\nabla\)-geodesic from \(p\) to \(q\) is straight in \(\theta\), \[ \theta(t)=(1-t)\theta(p)+t\theta(q), \] whereas the \(\nabla^*\)-geodesic is straight in \(\eta\). These paths need not coincide. Their tangent pairing is especially simple because \(g(\partial_{\theta^i},\partial_{\eta_j})=\delta_i^j\).
The exact three-point relation
Theorem. For any \(p,q,r\), \[ \mathcal D(p,r) =\mathcal D(p,q)+\mathcal D(q,r) +\{\theta(p)-\theta(q)\}^\top \{\eta(q)-\eta(r)\}. \]
Proof. Write all three canonical divergences as \(\psi(\theta_a)+\varphi(\eta_b)-\theta_a^\top\eta_b\). Potential terms cancel, leaving the displayed cross pairing.
Corollary (Pythagorean theorem). If the primal geodesic \(q\to p\) is orthogonal at \(q\) to the dual geodesic \(q\to r\), then \[ \mathcal D(p,r)=\mathcal D(p,q)+\mathcal D(q,r). \]
The cross term is exactly the metric pairing of the two initial tangent vectors because one is represented in \(\theta\)-coordinates and the other in \(\eta\)-coordinates.
Affine submanifolds and projection
Theorem. Let \(\mathcal C\) be closed and primal affine, and let \(q\) minimize \(\mathcal D(p,\cdot)\) over \(\mathcal C\) in the orientation matching the dual normal. Then the dual geodesic from \(q\) to \(p\) is orthogonal to \(T_q\mathcal C\), and for \(r\in\mathcal C\), \[ \mathcal D(p,r) =\mathcal D(p,q)+\mathcal D(q,r). \]
Proof. First-order optimality says the gradient of the objective at \(q\) annihilates every tangent direction in the affine set. Under Legendre duality that gradient is the dual-coordinate displacement from \(q\) to \(p\). Hence the required orthogonality holds, and the three-point theorem gives equality.
For merely convex sets, first-order optimality gives a nonnegative cross term rather than zero, producing a Pythagorean inequality. This distinction is the geometric difference between affine and convex constraints.