The final synthesis page records how information quantities enter statistical impossibility results without duplicating the statistical inference and decision theory notes.
Fano lower bound for estimation
Proposition. Let \(\theta_1,\ldots,\theta_N\) be \(2\delta\)-separated under a loss metric and let \(V\) be uniform on these indices. If data \(X\) are generated from \(P_{\theta_V}\), then every estimator has risk at least of order \(\delta\) whenever \[ I(V;X)\ll \log N. \]
Le Cam two-point principle
Proposition. For two separated parameters, testing affinity, total variation, Hellinger distance, or KL divergence controls the minimum estimation error. A typical form is \[ \inf_{\widehat\theta}\sup_{j=0,1}P_{\theta_j}\{d(\widehat\theta,\theta_j)\ge\delta\}\ge \frac12(1-\|P_{\theta_0}-P_{\theta_1}\|_{\mathrm{TV}}). \]
Experiments and information loss
Definition. A Markov transition from one experiment to another is a randomized reduction of observations. Data processing says such a reduction cannot increase pairwise divergences or mutual information. Equality corresponds to statistical sufficiency; approximate equality leads toward deficiency distance.
Connections to other notes
Remark. This page is a map rather than a replacement: LAN, minimax risk, deficiency, posterior contraction, and empirical-process entropy are developed in their own notes. Information theory supplies reusable inequalities and operational interpretations that those notes call upon.
Experiments as channels from parameters to data
Definition. A statistical experiment \(\mathcal E=(\mathcal X,\{P_\theta:\theta\in\Theta\})\) is a channel \(\theta\mapsto X\). Under prior \(\pi\), \[ I_\pi(\Theta;X)= \int D(P_\theta\|P_\pi)\,\pi(\mathrm{d}\theta), \qquad P_\pi=\int P_\theta\,\pi(\mathrm{d}\theta). \]
This is the expected information gained from the observation about the random parameter. It depends on the prior and is not by itself a frequentist information matrix.
Randomization of experiments loses information
Theorem. If experiment \(\mathcal F\) is obtained from \(\mathcal E\) by a parameter-free Markov kernel, then for every prior \[ I_\pi(\Theta;Y)\le I_\pi(\Theta;X), \] every pairwise \(f\)-divergence contracts, and every decision problem has Bayes risk under \(\mathcal E\) no larger than under \(\mathcal F\).
Proof. The Markov chain \(\Theta\to X\to Y\) gives information contraction. Pairwise divergence contraction is data processing. Any rule based on \(Y\) can be simulated by first generating \(Y\) from \(X\), so the rule class available under \(\mathcal E\) contains all randomized rules from \(\mathcal F\).
Le Cam deficiency
Definition. The deficiency of \(\mathcal E\) relative to \(\mathcal F\) is \[ \delta(\mathcal E,\mathcal F)= \inf_K\sup_{\theta} \|KP_\theta-Q_\theta\|_{\mathrm{TV}}, \] where \(K\) ranges over Markov kernels from the sample space of \(\mathcal E\) to that of \(\mathcal F\). The Le Cam distance is \[ \Delta(\mathcal E,\mathcal F)= \max\{\delta(\mathcal E,\mathcal F), \delta(\mathcal F,\mathcal E)\}. \]
Theorem. If losses are bounded by \(M\), then corresponding minimax and Bayes risks in two experiments differ by at most a constant multiple of \(M\Delta(\mathcal E,\mathcal F)\).
Proof outline. Approximately simulate data from one experiment using a near-optimal randomization kernel, then apply a decision rule from the other. Total variation bounds the change in expected bounded loss. Repeat in both directions.
Mutual information lower bounds for estimation
Theorem. Let prior \(\pi\) on parameter space have rate-distortion function \[ R_\pi(D)= \inf_{P_{\widehat\Theta\mid\Theta}: \mathbb Ed(\Theta,\widehat\Theta)\le D} I(\Theta;\widehat\Theta). \] For every estimator based on \(X\), \[ R_\pi\{\mathbb Ed(\Theta,\widehat\Theta)\} \le I_\pi(\Theta;X). \]
Proof. \(\Theta\to X\to\widehat\Theta\) gives \(I(\Theta;\widehat\Theta)\le I(\Theta;X)\). By definition, any reproduction achieving distortion \(D\) has information at least \(R_\pi(D)\).
Fano is recovered by choosing a uniform finite prior and a zero-one or packing distortion. Rate-distortion lower bounds can use the prior geometry more efficiently than exact decoding.
Bayesian Cramér-Rao and van Trees
Theorem. In a regular scalar model with prior density \(\pi\) vanishing suitably at boundaries, every estimator satisfies \[ \mathbb E(\widehat\theta-\theta)^2 \ge \frac1{ \mathbb E_\pi I_X(\theta)+I(\pi)}, \] where \[ I(\pi)=\int\frac{\{\pi'(\theta)\}^2}{\pi(\theta)}\,\mathrm{d}\theta. \]
Proof. Under the joint law, integrate by parts to obtain \[ \mathbb E[ (\widehat\theta-\theta) \{\partial_\theta\log p_\theta(X)+ \partial_\theta\log\pi(\theta)\}]=1. \] Cauchy-Schwarz bounds one by the product of mean-squared error and the joint score variance. Conditional score mean zero eliminates the cross term, leaving expected model Fisher information plus prior Fisher information.
I-MMSE in a Gaussian experiment
Theorem. For \(Y_\gamma=\sqrt\gamma\,\Theta+Z\), \[ I(\Theta;Y_\gamma) =\frac12\int_0^\gamma \mathbb E\{\Theta-\mathbb E(\Theta\mid Y_s)\}^2\,\mathrm{d}s. \]
This identity directly links information gained along signal-to-noise ratio with integrated Bayes estimation error. It is exact for any prior with finite second moment.
Local asymptotic normality
In regular iid models, local log likelihood ratios satisfy \[ \log\frac{\mathrm{d}P_{\theta_0+h/\sqrt n}^n} {\mathrm{d}P_{\theta_0}^n} =h^\top\Delta_n-\frac12h^\top I(\theta_0)h+o_P(1), \quad \Delta_n\Rightarrow N\{0,I(\theta_0)\}. \] The limit experiment is a Gaussian shift. Le Cam convergence transfers local minimax risk and testing limits, while KL between local alternatives converges to \(\tfrac12(h-k)^\top I(\theta_0)(h-k)\).
Sufficiency and posterior compression
Proposition. For prior \(\pi\) and statistic \(T(X)\), \[ I(\Theta;X)-I(\Theta;T) =I(\Theta;X\mid T) =\mathbb E_XD\{\pi(\cdot\mid X)\| \pi(\cdot\mid T)\}. \]
Proof. The first equality is the mutual-information chain rule because \(T\) is a function of \(X\). The second is conditional mutual information represented as expected KL between the full posterior and the posterior using only \(T\).
This gives an information measure of approximate sufficiency. Small average posterior KL does not automatically imply uniform decision equivalence; deficiency controls all bounded losses more directly.
Interfaces rather than duplication
Information theory supplies divergence contraction, testing exponents, and coding-style lower bounds. The full theories of minimax risk, tangent spaces, LAN, Bayes procedures, posterior contraction, and experiment comparison belong to the statistical inference, statistical decision theory, and Bayesian statistics notes. Cross-references preserve those distinctions while making the common information structure explicit.