Information spectrum methods keep the random information density instead of replacing it immediately by its expectation. This gives limit theorems beyond iid models.
Information density
Definition. For a pair \((X,Y)\) with joint law \(P_{XY}\), the information density is \[ i(X;Y)=\log\frac{P_{Y\mid X}(Y\mid X)}{P_Y(Y)}. \] For a sequence of pairs, the normalized information density is \(n^{-1}i(X^n;Y^n)\).
Spectral limits
Definition. The spectral inf-mutual information rate is the largest \(a\) such that \[ P\left\{\frac1n i(X^n;Y^n)<a\right\}\to0. \] The spectral sup version reverses the inequality. Similar definitions apply to entropy through \(-\log P_{X^n}(X^n)\).
Memoryless specialization
Proposition. For iid pairs \((X_i,Y_i)\), both spectral mutual information rates equal \(I(X;Y)\). For iid sources, both spectral entropy rates equal \(H(X)\).
Proof by concentration
Proof. In the iid case the normalized information density is an empirical average of integrable random variables. The law of large numbers collapses the spectrum to a point. For general non-iid sequences, the spectrum records the possible limiting range instead of forcing a single expectation.
Normalized information density
Definition. For arbitrary source laws \(P_{X^n}\), define \[ \imath_{X^n}(X^n)=-\log P_{X^n}(X^n). \] The spectral sup- and inf-entropy rates are \[ \overline H(\mathbf X)= p\text{-}\limsup_{n\to\infty} \frac1n\imath_{X^n}(X^n), \qquad \underline H(\mathbf X)= p\text{-}\liminf_{n\to\infty} \frac1n\imath_{X^n}(X^n). \]
Definition. For random variables \(Z_n\), \[ p\text{-}\limsup Z_n= \inf\{\alpha:P(Z_n>\alpha)\to0\}, \] and \[ p\text{-}\liminf Z_n= \sup\{\alpha:P(Z_n<\alpha)\to0\}. \]
For iid or stationary ergodic sources, the two rates coincide with Shannon entropy rate. Their separation measures failure of information stability.
General fixed-length source coding
Theorem. For an arbitrary source sequence \(\mathbf X=\{X^n\}\), the smallest asymptotically achievable fixed-length coding rate with vanishing error is \[ R^*=\overline H(\mathbf X). \]
Proof. For achievability, at rate \(R>\overline H\), encode the set \[ A_n=\{x^n:P_{X^n}(x^n)\ge e^{-nR}\}. \] It has at most \(e^{nR}\) elements and probability tending to one. For the converse, any codebook \(C_n\) of size \(e^{nR}\) satisfies, for \(\gamma>0\), \[ P(X^n\in C_n) \le P\{\imath_{X^n}(X^n)\le n(R+\gamma)\} +e^{-n\gamma}, \] because strings outside the low-information set each have probability below \(e^{-n(R+\gamma)}\). If \(R<\overline H\), success cannot tend to one.
Spectral mutual-information rates
Definition. For input-output pairs, \[ \imath_{X^n;Y^n}= \log\frac{\mathrm{d}P_{Y^n\mid X^n}} {\mathrm{d}P_{Y^n}}(Y^n\mid X^n), \] and \[ \underline I(\mathbf X;\mathbf Y)= p\text{-}\liminf\frac1n\imath_{X^n;Y^n}. \]
Verdú-Han channel formula
Theorem. For a general channel sequence \(W^n\), capacity under vanishing average error is \[ C=\sup_{\mathbf X}\underline I(\mathbf X;\mathbf Y), \] where the supremum ranges over input-process sequences satisfying any imposed constraints.
Proof outline. Random coding with threshold decoding succeeds when code rate lies below the probabilistic lower limit of normalized information density. Conversely, every code induces a uniform message and channel input; a meta-converse bounds success probability by the lower tail of information density plus an exponentially small slack. Optimizing over input sequences yields the formula.
A nonergodic mixture source
Example. First draw \(J\in\{1,2\}\), then generate the entire sequence iid from \(P_J\). Conditional AEP gives \[ -\frac1n\log P(X^n)\to H(P_J) \] in probability within each component. If \(H(P_1)<H(P_2)\), then \[ \underline H=H(P_1),\qquad \overline H=H(P_2). \] No single typical-set exponent carries nearly all probability for both components.
Second-order information spectrum
Definition. At first-order rate \(R\), second-order analysis studies quantiles of \[ \frac{\imath_{X^n}(X^n)-nR}{\sqrt n} \quad\text{or}\quad \frac{\imath_{X^n;Y^n}-nR}{\sqrt n}. \] For iid models these often converge normally, producing dispersion terms; information-spectrum definitions remain valid when no central limit theorem holds.
Smooth entropies
One-shot source coding is governed by quantiles rather than expectations. Smooth max-entropy measures the logarithmic size of a high-probability set after discarding error mass; hypothesis-testing entropy and smooth min-entropy govern related compression, randomness extraction, and cryptographic tasks. Their iid limits recover Shannon quantities through the asymptotic equipartition property.
Why expectation can be insufficient
Mutual information \(n^{-1}I(X^n;Y^n)\) is an average of information density. A small-probability region with enormous density can inflate the mean without supporting reliable communication for most messages. Operational coding theorems for arbitrary sources and channels therefore use probabilistic liminf and limsup rather than expected information alone.