Majorization-minimization replaces a difficult objective by a tractable surrogate that touches it at the current iterate. EM is the likelihood version of this principle, with Jensen's inequality and conditional distributions constructing the surrogate automatically.
Tangent majorants and monotonic descent
Definition. A function \(Q(\theta\mid\vartheta)\) majorizes \(f(\theta)\) at \(\vartheta\) if \[ Q(\theta\mid\vartheta)\ge f(\theta)\quad\text{for every }\theta, \qquad Q(\vartheta\mid\vartheta)=f(\vartheta). \] An MM step chooses \(\theta^+\) satisfying \(Q(\theta^+\mid\vartheta)\le Q(\vartheta\mid\vartheta)\).
MM descent. Every such step satisfies \(f(\theta^+)\le f(\vartheta)\). If equality can hold only when \(\vartheta\) is stationary, every nonstationary iteration strictly decreases the objective.
Proof. The two defining inequalities give \[ f(\theta^+) \le Q(\theta^+\mid\vartheta) \le Q(\vartheta\mid\vartheta) =f(\vartheta). \] The strict claim follows from the stated equality condition.
Monotonicity alone does not imply convergence to a global minimum. A surrogate should also agree with the first derivative of the objective at the touching point; otherwise an MM fixed point need not be stationary.
The likelihood-ELBO decomposition
Let \(X=x\) be observed, \(Z\) latent, and \(p_\theta(x,z)\) a joint density. For any density \(q(z)\), \[ \begin{aligned} \ell(\theta) &=\log p_\theta(x)\\ &=\underbrace{\mathbb E_q[\log p_\theta(x,Z)] -\mathbb E_q[\log q(Z)]}_{\mathcal L(q,\theta)} +\operatorname{KL}\!\left(q\,\middle\|\,p_\theta(\cdot\mid x)\right). \end{aligned} \]
Proof. Bayes' formula gives \(\log p_\theta(x,z) =\log p_\theta(z\mid x)+\log p_\theta(x)\). Substitute this into \(\mathcal L(q,\theta)\): \[ \mathcal L(q,\theta) =\log p_\theta(x) -\mathbb E_q\log\frac{q(Z)}{p_\theta(Z\mid x)}. \] The remaining expectation is the displayed KL divergence.
Because KL divergence is nonnegative, \(\mathcal L\) is an evidence lower bound. It is tight exactly when \(q=p_\theta(\cdot\mid x)\) almost everywhere.
Expectation-maximization as coordinate ascent
Definition. Starting from \(\theta_k\), EM performs
\[ \begin{aligned} \text{E-step:}\quad q_k(z)&=p_{\theta_k}(z\mid x),\\ \text{M-step:}\quad \theta_{k+1}&\in\operatorname*{arg\,max}_\theta \mathbb E_{q_k}[\log p_\theta(x,Z)]. \end{aligned} \]
Likelihood monotonicity. Every exact EM iteration satisfies \[ \ell(\theta_{k+1})\ge\ell(\theta_k). \] The same conclusion holds for a generalized EM step that merely increases the expected complete-data log likelihood.
Proof. Tightness at the E-step gives \(\ell(\theta_k)=\mathcal L(q_k,\theta_k)\). The M-step increases the only \(\theta\)-dependent term in the ELBO, so \[ \mathcal L(q_k,\theta_{k+1}) \ge\mathcal L(q_k,\theta_k). \] Finally, \(\ell(\theta_{k+1})\ge\mathcal L(q_k,\theta_{k+1})\) because the omitted KL divergence is nonnegative.
Score identity at an EM fixed point
Fisher identity. Under differentiation-under-the-integral regularity, \[ \nabla\ell(\theta) =\mathbb E_\theta[ \nabla_\theta\log p_\theta(X,Z)\mid X=x]. \] Consequently an interior EM fixed point that exactly maximizes the M-step objective is a stationary point of the observed log-likelihood.
Proof. Differentiate \(p_\theta(x)=\int p_\theta(x,z)\,\mathrm{d}z\): \[ \nabla\log p_\theta(x) =\int\frac{\nabla p_\theta(x,z)}{p_\theta(x)}\,\mathrm{d}z =\int\nabla\log p_\theta(x,z) p_\theta(z\mid x)\,\mathrm{d}z. \] At a fixed point, first-order optimality of the M-step makes this conditional expectation zero.
A Gaussian-mixture E-step
Example. In a \(K\)-component Gaussian mixture, \[ p_\theta(x_i)=\sum_{k=1}^K\pi_k\varphi(x_i;\mu_k,\Sigma_k). \] The E-step computes responsibilities \[ \tau_{ik} =\frac{\pi_k\varphi(x_i;\mu_k,\Sigma_k)} {\sum_{\ell=1}^K\pi_\ell \varphi(x_i;\mu_\ell,\Sigma_\ell)}. \] The M-step replaces unobserved component indicators by these conditional means, giving weighted updates such as \(\mu_k^+=\sum_i\tau_{ik}x_i/\sum_i\tau_{ik}\). The likelihood increases, but different initializations can converge to different local maxima or singular configurations.
The missing-data and likelihood formulation is developed further in missing data and EM; the variational interpretation becomes approximate when the E-step posterior is restricted to a tractable family.