Mixture geometry uses probability distributions themselves as affine objects. A mixture family is flat for the mixture connection because convex combinations are straight.
Mixture families
A mixture family is linear in densities or probability masses.
Definition. A finite-dimensional mixture family has densities \[ p_\eta(x)=p_0(x)+\eta^aF_a(x) \] on the domain where this expression is positive and normalized. Equivalently, it may be the convex hull of fixed component distributions.
Remark. Mixture coordinates are expectation-like coordinates because linear constraints on probabilities are affine in them.
Mixture geodesics
Straightness in mixture geometry means ordinary convex interpolation of distributions.
Proposition. The curve \(p_t=(1-t)p_0+tp_1\) is a mixture geodesic wherever it remains in the positive model.
Proof. In mixture-affine coordinates, \(p_t\) is a straight line: its second derivative in those coordinates is zero. Therefore its acceleration under the mixture connection vanishes.
Duality with exponential families
Mixture-flat and exponential-flat submanifolds meet through KL Pythagorean identities.
Theorem. In a dually flat model, if a mixture-flat submanifold and an exponential-flat submanifold meet orthogonally, then KL divergence decomposes by the generalized Pythagorean theorem.
Proof. Mixture-flat and exponential-flat mean that the relevant connecting curves are geodesics for dual connections. Orthogonality under the Fisher metric cancels the three-point term in the canonical divergence identity, giving the KL decomposition.
Statistics of mixtures
Mixture models in the ordinary statistical sense can be singular even though mixture geometry is affine.
Remark. Finite mixtures with unknown components often have label switching and nonidentifiability. The mixture connection is simple for convex combinations of fixed distributions, but mixture-model parameter spaces can be singular.
Affine mixtures
Definition. Given densities \(p_0,\ldots,p_k\), the mixture family is \[ p_\eta=p_0+\sum_{a=1}^k\eta_a(p_a-p_0), \] on the open coefficient domain where \(p_\eta>0\) and integrates to one. Because \(p_\eta\) is affine in \(\eta\), these coordinates are affine for the mixture connection.
The tangent scores are \[ \ell_a(x)=\frac{p_a(x)-p_0(x)}{p_\eta(x)}, \] and the Fisher matrix is \[ g_{ab}(\eta) =\int\frac{\{p_a-p_0\}\{p_b-p_0\}}{p_\eta}\,\mathrm{d}\mu. \] Linear independence of the component differences in \(L_1\) is required for a regular affine family.
Dual exponential coordinates
On the full finite simplex, mixture coordinates are probabilities and their dual exponential coordinates are log odds. More generally, a mixture-flat submanifold has an exponential-flat normal family. Moment constraints and exponential tilts therefore meet orthogonally in the canonical dual geometry, which underlies KL Pythagorean identities.
Finite mixtures are not globally regular mixture families
The statistical model \[ p(x)=\sum_{j=1}^k\alpha_j f(x;\xi_j) \] is affine in weights when component parameters \(\xi_j\) are fixed. Once the locations or scales also vary, it is no longer a mixture-affine subspace. Moreover, zero weights, coincident components, and label permutations create singularities.
Example (vanishing component). In a two-normal mixture, at \(\alpha=0\) the parameter \(\mu_1\) has no effect on the law: \[ \left.\frac{\partial}{\partial\mu_1} \{\alpha\phi(x-\mu_1)+(1-\alpha)\phi(x-\mu_2)\} \right|_{\alpha=0}=0. \] The corresponding score vanishes and Fisher information loses rank. Root-\(n\) regular asymptotics and smooth-manifold geometry both fail there.
Expectation--maximization as alternating geometry
Introduce latent labels \(Z\) and a complete-data model \(Q_\vartheta(X,Z)\). The EM algorithm alternates between:
- choosing a joint law with observed marginal fixed and conditional \(Z\mid X\) equal to the current posterior;
- projecting that joint law onto the complete-data model by maximizing expected log likelihood.
The KL chain rule proves monotonicity: \[ \ell(\vartheta^{t+1})-\ell(\vartheta^t) \ge Q(\vartheta^{t+1}\mid\vartheta^t) -Q(\vartheta^t\mid\vartheta^t)\ge0. \] This alternating-projection picture is exact in lifted distribution space, though nonconvexity of the parametric mixture model still permits local maxima and singular limits.