Gaussian distributions are extremal for entropy under covariance constraints. This extremality yields the capacity of additive Gaussian channels.
Maximum entropy at fixed covariance
Theorem. If \(X\in\mathbb R^d\) has covariance matrix \(\Sigma\), then \[ h(X)\le \frac12\log\big((2\pi e)^d\det\Sigma\big), \] with equality if and only if \(X\) is Gaussian with covariance \(\Sigma\).
AWGN capacity
Theorem. For the additive channel \(Y=X+Z\), where \(Z\sim N(0,N)\) is independent noise and \(\mathbb E X^2\le P\), the capacity per use is \[ C=\frac12\log\left(1+\frac PN\right). \]
Proof by entropy bound
Proof. For any input satisfying the power constraint, \[ I(X;Y)=h(Y)-h(Z)\le \frac12\log(2\pi e(P+N))-\frac12\log(2\pi eN). \] Equality is attained by Gaussian \(X\sim N(0,P)\), independent of \(Z\).
Role of Gaussianity
Remark. The theorem is not saying Gaussian noise is easiest; it says that once the noise is Gaussian and the input power is fixed, Gaussian input maximizes output entropy and therefore mutual information.
Gaussian maximum-entropy theorem
Theorem. Among all distributions on \(\mathbb R^d\) with mean \(\mu\) and covariance \(\Sigma\succ0\), \[ h(X)\le\frac12\log\{(2\pi e)^d\det\Sigma\}, \] with equality exactly for \(N_d(\mu,\Sigma)\).
Proof. Let \(G\) have the matching Gaussian law. Nonnegativity of KL gives \[ 0\le D(P_X\|P_G) =-h(X)-\mathbb E\log g(X). \] The Gaussian log density is a constant plus a quadratic form, whose expectation depends only on the common mean and covariance. Hence \(-\mathbb E\log g(X)=h(G)\). Equality in KL forces \(X\sim G\).
Scalar AWGN capacity
Theorem. For \[ Y=X+Z,\qquad Z\sim N(0,N), \] independent of \(X\), under power constraint \(\mathbb EX^2\le P\), \[ C=\frac12\log\left(1+\frac PN\right) \] nats per channel use, achieved by \(X\sim N(0,P)\).
Proof. \[ I(X;Y)=h(Y)-h(Z). \] Since \(\operatorname{Var}(Y)\le P+N\), Gaussian extremality gives \(h(Y)\le\tfrac12\log\{2\pi e(P+N)\}\). Subtract \(h(Z)=\tfrac12\log(2\pi eN)\). Gaussian \(X\) makes \(Y\) Gaussian with variance \(P+N\), attaining equality.
Parallel Gaussian channels and water filling
Theorem. For independent subchannels \(Y_i=X_i+Z_i\), \(Z_i\sim N(0,N_i)\), with \(\sum_iP_i\le P\), capacity is \[ C=\max_{P_i\ge0,\sum P_i\le P} \frac12\sum_i\log\left(1+\frac{P_i}{N_i}\right), \] and the optimizer is \[ P_i=(\nu-N_i)_+ \] for a water level \(\nu\) satisfying the power constraint.
Proof. Independent Gaussian inputs are optimal by vector Gaussian extremality and diagonal noise covariance. KKT stationarity for active coordinates gives \[ \frac1{2(N_i+P_i)}=\lambda, \] so \(P_i=\nu-N_i\) with \(\nu=1/(2\lambda)\). Complementary slackness sets negative values to zero.
Vector Gaussian channel
Theorem. For \(Y=HX+Z\), \(Z\sim N(0,\Sigma_Z)\), and covariance constraint \(\operatorname{tr}\Sigma_X\le P\), \[ C= \max_{\Sigma_X\succeq0,\operatorname{tr}\Sigma_X\le P} \frac12\log\det\left( I+\Sigma_Z^{-1/2}H\Sigma_XH^\top \Sigma_Z^{-1/2}\right). \]
Proof outline. Gaussian maximum entropy makes a Gaussian input optimal for every covariance. Whiten noise and diagonalize \(H^\top\Sigma_Z^{-1}H\); the determinant becomes a product of scalar subchannel factors. Water fill over its eigenmodes.
Gaussian rate-distortion preview
Proposition. For \(X\sim N(0,\sigma^2)\) and squared-error distortion, \[ R(D)= \frac12\log\frac{\sigma^2}{D}, \quad0<D<\sigma^2, \] and \(R(D)=0\) for \(D\ge\sigma^2\).
The optimal test channel is Gaussian and can be represented backward as \(X=\widehat X+Z_D\), with \(Z_D\sim N(0,D)\) independent of \(\widehat X\sim N(0,\sigma^2-D)\).
Worst additive noise lemma
Theorem. Under covariance constraints, Gaussian noise minimizes mutual information \(I(X;X+Z)\) for Gaussian \(X\) among independent noises with the same covariance.
Proof outline. Compare the target noise with a Gaussian through relative entropy and use data processing under convolution with \(X\). The difference in divergences rearranges to the difference of mutual informations; Gaussian matching of second moments controls the cross-entropy terms.
Gaussian extremality has hypotheses
Gaussian laws maximize entropy under covariance, not under every constraint. Peak-amplitude constraints, discreteness, sparsity, or nonlinear channel laws can produce non-Gaussian capacity-achieving inputs. In an amplitude-constrained Gaussian channel, the optimal input is discrete with finitely many mass points.