The data-processing principle says that stochastic postprocessing cannot create information. Equality is the mathematical form of lossless reduction.
Markov kernels
Definition. A channel or Markov kernel \(K\) sends an input law \(P\) to the output law \(PK\). A Markov chain \(U\to X\to Y\) means the conditional law of \(Y\) given \((U,X)\) depends only on \(X\).
Data-processing inequality
Theorem. If \(U\to X\to Y\), then \[ I(U;Y)\le I(U;X). \] Equivalently, for every Markov kernel \(K\), \[ D(PK\|QK)\le D(P\|Q). \]
Proof by conditional mutual information
Proof. The chain rule gives \[ I(U;X,Y)=I(U;X)+I(U;Y\mid X)=I(U;Y)+I(U;X\mid Y). \] The Markov condition makes \(I(U;Y\mid X)=0\). Hence \(I(U;X)-I(U;Y)=I(U;X\mid Y)\ge0\).
Sufficiency as equality
Proposition. For a family \(\{P_\theta\}\), a statistic \(T(X)\) is information-lossless for pairwise discrimination if \[ D(P_\theta\|P_{\theta'})=D(P_\theta T^{-1}\|P_{\theta'}T^{-1}) \] for all \(\theta,\theta'\). In dominated finite models, this is equivalent to the usual factorization form of sufficiency.
Data-processing inequality
Theorem. If \(X\to Y\to Z\) is a Markov chain, then \[ I(X;Z)\le I(X;Y). \]
Proof. By the chain rule, \[ I(X;Y,Z)=I(X;Y)+I(X;Z\mid Y)=I(X;Y) \] because the Markov property makes the conditional information zero. Also \[ I(X;Y,Z)=I(X;Z)+I(X;Y\mid Z)\ge I(X;Z). \]
Divergence contraction under a channel
Theorem. For a Markov kernel \(K\), \[ D(PK\|QK)\le D(P\|Q). \]
Proof. Define joint laws \(P_{XY}=P_XK_{Y\mid X}\) and \(Q_{XY}=Q_XK_{Y\mid X}\). The KL chain rule gives \(D(P_{XY}\|Q_{XY})=D(P\|Q)\) because conditional kernels agree. Reversing the chain rule, \[ D(P_{XY}\|Q_{XY}) =D(PK\|QK)+ \mathbb E_{PK}D(P_{X\mid Y}\|Q_{X\mid Y}) \ge D(PK\|QK). \]
Equality and reverse kernels
Theorem. Equality \(D(PK\|QK)=D(P\|Q)\) holds exactly when \[ P_{X\mid Y}=Q_{X\mid Y} \quad PK\text{-almost surely}, \] under common-support conditions. Equivalently, there is a reverse kernel \(L\) such that \(PKL=P\) and \(QKL=Q\).
Proof. Equality in the preceding chain-rule decomposition is equivalent to zero conditional KL, hence equal reverse conditional laws. Taking this common conditional law as \(L\) reconstructs both inputs. Conversely, a common reverse kernel and two applications of data processing give \[ D(P\|Q)=D(PKL\|QKL)\le D(PK\|QK)\le D(P\|Q). \]
Statistical sufficiency
Definition. A statistic \(T(X)\) is sufficient for a family \(\{P_\theta\}\) if the conditional law of \(X\) given \(T\) does not depend on \(\theta\).
Theorem. In a dominated family, \(T\) is sufficient if and only if the density factorizes as \[ p_\theta(x)=g_\theta\{T(x)\}h(x). \]
Proof outline. If factorization holds, normalization within each fiber of \(T\) cancels \(g_\theta\), so the conditional law is parameter-free. Conversely, choose a reference conditional density on each fiber and write the joint density as the marginal density of \(T\), which carries all \(\theta\)-dependence, times this common conditional density.
Corollary. If \(T\) is sufficient, then for every pair \(\theta,\theta'\), \[ D(P_\theta^T\|P_{\theta'}^T) =D(P_\theta\|P_{\theta'}). \]
Bayesian sufficiency and information equality
Proposition. Let \(\Theta\sim\pi\) and \(X\sim P_\Theta\). For any statistic \(T=T(X)\), \[ I(\Theta;T)\le I(\Theta;X). \] Equality holds exactly when \(\Theta\perp\!\!\!\perp X\mid T\), that is, when the posterior based on \(X\) depends on \(X\) only through \(T\).
Proof. Since \(\Theta\to X\to T\), data processing gives the inequality. The information loss is \[ I(\Theta;X)-I(\Theta;T)=I(\Theta;X\mid T), \] which vanishes exactly at conditional independence.
Blackwell comparison of experiments
Definition. Experiment \(\mathcal E=\{P_\theta\}\) Blackwell-dominates \(\mathcal F=\{Q_\theta\}\) if there is a parameter-free Markov kernel \(K\) with \(Q_\theta=P_\theta K\) for every \(\theta\).
Every decision rule in \(\mathcal F\) can then be simulated from \(\mathcal E\), so \(\mathcal E\) has no larger Bayes risk for every prior and loss. Pairwise divergence contraction is necessary for dominance but generally not sufficient; Blackwell order is a simultaneous decision-theoretic comparison.
Strong data-processing inequalities
Definition. A channel \(K\) has KL contraction coefficient \[ \eta_{\mathrm{KL}}(K)= \sup_{P\ne Q}\frac{D(PK\|QK)}{D(P\|Q)}\le1. \] If \(\eta_{\mathrm{KL}}<1\), information decays quantitatively: \[ I(U;Y)\le\eta_{\mathrm{KL}}I(U;X) \] for \(U\to X\to Y\), under the corresponding input restrictions.
For a binary symmetric channel with crossover probability \(\varepsilon\), contraction is strict unless \(\varepsilon\in\{0,1\}\). Strong inequalities quantify how repeated noisy channels erase information and connect to mixing, functional inequalities, and privacy.