Mutual information measures dependence by comparing the joint law with the product of its marginals. Conditional mutual information is the corresponding obstruction to conditional independence.
Mutual information
Definition. For finite \(X,Y\), \[ I(X;Y)=D(P_{XY}\|P_XP_Y)=H(X)+H(Y)-H(X,Y). \] The conditional mutual information is \[ I(X;Y\mid Z)=H(X\mid Z)-H(X\mid Y,Z). \]
Chain rules
Proposition. For random variables \(X_1,\ldots,X_n,Y,Z\), \[ I(X_1,\ldots,X_n;Y\mid Z)=\sum_{i=1}^n I(X_i;Y\mid Z,X_1,\ldots,X_{i-1}). \] Also \(I(X;Y\mid Z)\ge0\), with equality if and only if \(X\) and \(Y\) are conditionally independent given \(Z\).
Proof from relative entropy
Proof. The identity \(I(X;Y)=D(P_{XY}\|P_XP_Y)\) makes nonnegativity immediate. Conditional mutual information is the average of \(D(P_{XY\mid Z=z}\|P_{X\mid Z=z}P_{Y\mid Z=z})\) over \(z\). The chain rule follows by expanding conditional entropies and canceling adjacent terms.
Independence certificate
Remark. The equation \(I(X;Y\mid Z)=0\) is often more useful than the factorization itself: it turns conditional independence into an inequality statement stable under limiting and convexity arguments.
Equivalent representations
Theorem. For discrete \(X,Y\), \[ \begin{aligned} I(X;Y) &=D(P_{XY}\|P_XP_Y)\\ &=H(X)-H(X\mid Y)\\ &=H(Y)-H(Y\mid X)\\ &=H(X)+H(Y)-H(X,Y). \end{aligned} \]
Proof. Expand \[ \log\frac{p(x,y)}{p(x)p(y)} =\log p(x,y)-\log p(x)-\log p(y) \] and integrate. Rearranging with the entropy chain rule gives the other forms.
Corollary. \(I(X;Y)\ge0\), with equality exactly when \(X\) and \(Y\) are independent.
Information density
Definition. The information density is \[ \imath_{X;Y}(x;y)= \log\frac{p_{Y\mid X}(y\mid x)}{p_Y(y)} =\log\frac{p_{XY}(x,y)}{p_X(x)p_Y(y)}. \] Mutual information is its expectation. Information density can be negative for particular pairs even though its mean is nonnegative.
Conditional mutual information
Theorem. \[ I(X;Y\mid Z) =\mathbb E_ZD(P_{XY\mid Z}\| P_{X\mid Z}P_{Y\mid Z}) =H(X\mid Z)-H(X\mid Y,Z)\ge0. \] Equality holds exactly when \(X\perp\!\!\!\perp Y\mid Z\).
Proof. Apply the mutual-information identity conditionally at each \(Z=z\) and average. Nonnegativity and equality follow from conditional KL divergence.
Chain rules
Theorem. \[ I(X;Y,Z)=I(X;Z)+I(X;Y\mid Z), \] and more generally \[ I(X;Y_1,\ldots,Y_n) =\sum_{i=1}^nI(X;Y_i\mid Y_1,\ldots,Y_{i-1}). \]
Proof. Write \(I(X;Y,Z)=H(X)-H(X\mid Y,Z)\), add and subtract \(H(X\mid Z)\), and iterate.
Mutual information as expected posterior change
Proposition. \[ I(X;Y)= \mathbb E_YD(P_{X\mid Y}\|P_X) =\mathbb E_XD(P_{Y\mid X}\|P_Y). \]
Proof. Disintegrate the joint KL \(D(P_{XY}\|P_XP_Y)\) first with respect to \(Y\), then with respect to \(X\), using the relative-entropy chain rule.
Thus information is the expected KL movement from prior to posterior, or equivalently the expected distinguishability of a conditional output law from the output mixture.
Entropy bounds
Proposition. \[ I(X;Y)\le\min\{H(X),H(Y)\}. \] Equality \(I(X;Y)=H(X)\) holds exactly when \(X\) is a function of \(Y\) almost surely.
Proof. \(I(X;Y)=H(X)-H(X\mid Y)\le H(X)\), and symmetrically. Equality means \(H(X\mid Y)=0\).
Interaction information can have either sign
Definition. The three-variable interaction information is \[ I(X;Y;Z)=I(X;Y)-I(X;Y\mid Z), \] a symmetric quantity that can be positive or negative.
Example. Let \(X,Y\) be independent fair bits and \(Z=X\oplus Y\). Then \(I(X;Y)=0\), but given \(Z\), one bit determines the other, so \(I(X;Y\mid Z)=\log2\). Hence \(I(X;Y;Z)=-\log2\), expressing synergistic information not present in any pair alone.
Total correlation
Definition. For \(X_1,\ldots,X_n\), total correlation is \[ \operatorname{TC}(X_1,\ldots,X_n) =D\!\left(P_{X_1,\ldots,X_n}\middle\| \prod_iP_{X_i}\right) =\sum_iH(X_i)-H(X_1,\ldots,X_n). \] It is nonnegative and vanishes exactly at mutual independence.
Gaussian mutual information
Example. If \((X,Y)\) is jointly Gaussian with correlation \(\rho\), then \[ I(X;Y)=-\frac12\log(1-\rho^2). \] More generally, for Gaussian vectors, \[ I(X;Y)=\frac12\log \frac{\det\Sigma_X\det\Sigma_Y}{\det\Sigma_{XY}}. \] This follows by inserting Gaussian differential entropies; unlike correlation, mutual information detects arbitrary dependence outside the Gaussian family.