Information projections minimize a divergence over a model or constraint set. Their first-order conditions are geometric orthogonality conditions.
I- and M-projections
The names refer to the two KL directions and their relation with exponential and mixture geometry.
Definition. For a set \(\mathcal C\), an I-projection often means a minimizer of \(D(P\|Q)\) or \(D(Q\|P)\) depending on convention. To avoid ambiguity, this note always states the objective explicitly: \[ \min_{Q\in\mathcal C}D(P\|Q),\qquad \min_{Q\in\mathcal C}D(Q\|P). \]
Remark. The ambiguity of terminology is common. The geometry is unambiguous once the KL order and the affine structure of \(\mathcal C\) are specified.
Moment constraints and exponential form
Minimizing KL subject to moment constraints gives exponential tilting.
Theorem. Minimizing \(D(Q\|P)\) subject to \(\mathbb E_Q T_j=c_j\) gives, when a solution exists, \[ \frac{\mathrm{d}Q_\lambda}{\mathrm{d}P} =\exp\{\lambda^jT_j-\psi(\lambda)\}. \]
Proof. Use Lagrange multipliers for normalization and moment constraints. The first variation of \(\int q\log(q/p)\,\mathrm{d}\mu\) plus linear constraint terms gives \(\log(q/p)+1-\alpha-\lambda^jT_j=0\). Solving yields the exponential tilt, with \(\psi\) normalizing the density.
Projection onto exponential families
Maximum likelihood can be read as a KL projection in expectation coordinates.
Proposition. For an exponential family, the likelihood equations set model expectation parameters equal to empirical sufficient-statistic averages.
Proof. The log likelihood is \(\theta^i\bar T_i-\psi(\theta)\) up to constants. Differentiating gives \(\bar T_i-\partial_i\psi(\theta)\). Setting this to zero yields \(\eta_i(\theta)=\bar T_i\).
Alternating projections
Iterative proportional fitting alternates between dual affine constraint sets.
Remark. When constraint sets are mixture-flat or exponential-flat, alternating KL projections can be analyzed through repeated Pythagorean identities.
Existence and uniqueness
An information projection is a minimizer of a directed divergence over a model or constraint set. In finite dimensions, continuity plus compactness gives existence; strict convexity in the optimized argument gives uniqueness. On noncompact or infinite-dimensional sets, one instead needs coercivity, uniform integrability, or lower semicontinuity in an appropriate topology.
Theorem (linear-family I-projection). Let \[ \mathcal C=\left\{Q:E_QT_j=c_j,\ 1\le j\le k\right\} \] be nonempty, and let \(P\) have positive density. Under an interior constraint qualification, the minimizer of \(D(Q\|P)\) over \(\mathcal C\) has \[ \frac{\mathrm{d}Q_\lambda}{\mathrm{d}P}(x) =\exp\{\lambda^\top T(x)-\psi_P(\lambda)\}, \] where \(\lambda\) is chosen to match the moments.
Proof. Introduce multipliers for normalization and moment constraints. The first variation of \(\int q\log(q/p)\,\mathrm{d}\mu\) in a signed direction \(h\) is \(\int h\{1+\log(q/p)\}\,\mathrm{d}\mu\). Stationarity therefore requires \(\log(q/p)=\lambda^\top T-\text{constant}\). Normalization gives \(\psi_P\), and strict convexity of KL in \(Q\) gives uniqueness.
Corollary. For every \(Q\in\mathcal C\), \[ D(Q\|P)=D(Q\|Q_\lambda)+D(Q_\lambda\|P). \]
Proof. Subtract the three KL terms. The remainder is \[ \int(q-q_\lambda)\log(q_\lambda/p)\,\mathrm{d}\mu, \] which vanishes because \(\log(q_\lambda/p)\) is an affine combination of the constrained moments.
Projection onto an exponential family
For an exponential family \(Q_\theta\), minimizing \(D(P\|Q_\theta)\) matches \(E_PT\) to \(E_{Q_\theta}T\). This is the population form of maximum likelihood: replacing \(P\) by the empirical distribution makes the M-projection equation the likelihood score equation. Failure of the empirical moment to lie in the interior of the mean space explains nonexistence of a finite MLE.
Alternating projections
Theorem (finite-dimensional Bregman alternation). For closed affine sets with nonempty intersection, cyclic Bregman projections decrease the divergence to the intersection and, under standard Legendre and coercivity conditions, converge to its projection.
Iterative proportional fitting is the KL instance for marginal constraints. Each step rescales one margin, satisfies an exact Pythagorean identity, and cannot increase the target divergence. For general convex sets, Dykstra corrections are needed; naive alternation need not recover the joint projection.