Separation theorems say that source representation and channel transmission can be optimized separately under the classical iid assumptions.
Lossless separation
Theorem. For an iid source with entropy \(H(X)\) and a discrete memoryless channel of capacity \(C\), reliable transmission is possible if \(H(X)<C\) and impossible if \(H(X)>C\).
Lossy separation
Theorem. For distortion level \(D\), lossy communication is possible when \(R(D)<C\) and impossible when \(R(D)>C\), under the standard memoryless source and channel assumptions.
Proof by concatenation and converse
Proof. The direct part concatenates a source code of rate below channel capacity with a channel code of rate above the source-coding requirement. The converse combines the channel upper bound \(I\le nC\) with the source lower bound \(I\ge nR(D)\), after single-letterization.
Scope of the theorem
Remark. Separation is a theorem with assumptions, not a universal design law. Dependence, delay constraints, multi-terminal settings, or semantic constraints can break the clean source/channel split.
Separated architecture
A separated system first compresses the source to a bitstream near its rate-distortion limit, then protects those bits with a channel code near channel capacity. The theorem says that, for memoryless point-to-point asymptotics and compatible criteria, this modular architecture loses no first-order performance.
Separation theorem
Theorem. Let a memoryless source produce \(k\) symbols and a DMC be used \(n\) times. Put \(\rho=k/n\) source symbols per channel use. Distortion \(D\) is achievable if \[ \rho R(D)<C. \] Conversely, every achievable \(D\) must satisfy \[ \rho R(D)\le C. \]
Achievability
Proof. Choose rates \[ R(D)<R_s<R_c/\rho<C/\rho. \] A rate-distortion code compresses \(k\) source symbols into about \(e^{kR_s}\) indices with distortion at most \(D+\varepsilon\). This index set requires channel rate \(kR_s/n=\rho R_s<C\), so a channel code transmits it with vanishing error. On successful channel decoding, source distortion is controlled by the source code; the rare channel error contributes negligibly for bounded distortion or under suitable moment control.
Converse
Proof. Let \(\widehat S^k\) be the final reconstruction. Rate-distortion converse gives \[ kR(D)\le I(S^k;\widehat S^k). \] The communication chain \(S^k\to X^n\to Y^n\to\widehat S^k\) and data processing give \[ I(S^k;\widehat S^k)\le I(X^n;Y^n)\le nC. \] Divide by \(n\) to obtain \(\rho R(D)\le C\).
Lossless separation
Corollary. For lossless transmission, \(R(0)=H(S)\), so reliable communication is possible exactly when \[ \rho H(S)<C \] at first order.
Bandwidth mismatch
If there are \(\kappa\) channel uses per source symbol, then \(\rho=1/\kappa\) and the condition becomes \[ R(D)<\kappa C. \] For a Gaussian source and AWGN channel this yields \[ D\ge\sigma_S^2 \left(1+\frac PN\right)^{-\kappa}. \]
Why interfaces matter
Separation permits independently designed source and channel modules because the compressed bits are asymptotically almost uniform and the channel code only needs their rate. The theorem does not say finite-blocklength latency, complexity, or error propagation are unaffected by modular design.
Second-order mismatch
At finite blocklength, source dispersion and channel dispersion combine through error allocation. Separately optimizing each module at an arbitrary split of total error can be suboptimal relative to joint source-channel coding, even though first-order thresholds agree.
When separation can fail
- multiuser networks with correlated sources and interference;
- strict delay or zero-delay constraints;
- channels with unknown states where source side information interacts with state;
- semantic or task losses not represented by a separable single-letter distortion;
- finite-blocklength systems where modular overhead is substantial;
- networks requiring coordination rather than point-to-point reproduction.
Uncoded Gaussian transmission
Example. With one channel use per sample, send \(X_c=\sqrt{P/\sigma_S^2}\,S\) over AWGN and estimate linearly. The resulting distortion is \[ D=\frac{\sigma_S^2}{1+P/N}, \] exactly the separation limit for \(\kappa=1\). This exceptional equality reflects matching Gaussian and quadratic structures; uncoded transmission is not generally optimal.
Joint source-channel error exponents
Below the separation threshold, the best end-to-end excess-distortion exponent need not equal the minimum of independently optimized source and channel exponents. Joint codes can balance atypical source blocks against channel noise more efficiently. First-order separation and exponent-level separation are different theorems.