For dependent sequences, the central object is entropy per symbol. Stationarity restores enough structure for limits to exist.
Entropy rate
Definition. For a discrete stochastic process \(X_1,X_2,\ldots\), define \[ \overline H_n=\frac1n H(X_1,\ldots,X_n),\qquad h_n=H(X_n\mid X_1, \ldots,X_{n-1}). \] When the limit exists, the entropy rate is \(\overline H=\lim_n \overline H_n\).
Stationary limit theorem
Theorem. If the process is stationary, then \(h_n\) decreases to a limit and \[ \lim_{n\to\infty}\frac1nH(X_1^n)=\lim_{n\to\infty}H(X_n\mid X_1^{n-1}). \]
Proof by monotonicity
Proof. Conditioning reduces entropy, so \(H(X_n\mid X_1^{n-1})\) is monotone after rewriting by stationarity. The block entropy satisfies \[ H(X_1^n)=\sum_{i=1}^n H(X_i\mid X_1^{i-1}), \] so \(n^{-1}H(X_1^n)\) is the Cesàro average of the conditional entropies and has the same limit.
Markov chains
Corollary. If \((X_n)\) is a stationary Markov chain with transition matrix \(P(y\mid x)\) and stationary law \(\pi\), then \[ \overline H=\sum_x\pi(x)H(P(\cdot\mid x)). \]
Block entropy and entropy rate
Definition. For a discrete process \((X_t)\), let \[ H_n=H(X_1,\ldots,X_n). \] The entropy rate, when it exists, is \[ \overline H(X)=\lim_{n\to\infty}\frac{H_n}{n}. \]
Existence under stationarity
Theorem. For a stationary finite-alphabet process, \[ \overline H(X)=\inf_{n\ge1}\frac{H_n}{n} =\lim_{n\to\infty}H(X_n\mid X_1,\ldots,X_{n-1}). \]
Proof. Stationarity and subadditivity give \(H_{m+n}\le H_m+H_n\). Fekete's lemma yields \(\lim H_n/n=\inf H_n/n\). Let \(a_n=H(X_n\mid X_1^{n-1})\). By stationarity, \[ a_{n+1}=H(X_{n+1}\mid X_1^n) \le H(X_{n+1}\mid X_2^n)=a_n, \] so \(a_n\downarrow a\). The chain rule gives \(H_n/n=n^{-1}\sum_{i=1}^na_i\to a\) by Cesàro convergence.
Conditioning on the infinite past
Corollary. For a two-sided stationary process, \[ \overline H(X)= H(X_0\mid X_{-1},X_{-2},\ldots), \] where conditional entropy is the limit along increasing finite pasts.
Markov-chain entropy rate
Theorem. If \(X_t\) is a stationary Markov chain with invariant law \(\pi\) and transition matrix \(P\), then \[ \overline H(X)=H(X_1\mid X_0) =-\sum_i\pi_i\sum_jP_{ij}\log P_{ij}. \]
Proof. The Markov property gives \[ H(X_n\mid X_1^{n-1})=H(X_n\mid X_{n-1}). \] Stationarity makes this quantity independent of \(n\). Expand conditional entropy under the stationary joint law \(\pi_iP_{ij}\).
Example. For a binary symmetric Markov chain with flip probability \(q\) and stationary law \((1/2,1/2)\), \[ \overline H=h_2(q). \] Marginal entropy is \(\log2\), but predictable persistence reduces new information per symbol.
Deterministic processes
Example. Let \(X_t=X_0\) for all \(t\), with \(X_0\) a fair bit. Then \(H_n=\log2\) for every \(n\), so \(\overline H=0\), despite \(H(X_t)=\log2\). Entropy rate measures new uncertainty per time step, not marginal randomness.
Mutual-information rate
Definition. For jointly stationary processes, \[ \overline I(X;Y)= \lim_{n\to\infty}\frac1nI(X_1^n;Y_1^n) \] when the limit exists. Since \[ I(X_1^n;Y_1^n)=H(X_1^n)+H(Y_1^n)-H(X_1^n,Y_1^n), \] existence follows whenever the three entropy rates exist.
Directed information for feedback
Definition. Directed information from \(X^n\) to \(Y^n\) is \[ I(X^n\to Y^n) =\sum_{t=1}^nI(X^t;Y_t\mid Y^{t-1}). \]
Ordinary mutual information is symmetric and cannot distinguish direction under feedback. Directed information measures the information flowing causally from past and present inputs to the current output and appears in feedback-channel capacity.
Excess entropy
Definition. When finite, \[ \mathcal E= \lim_{n\to\infty}\{H_n-n\overline H\} =I(X_{-\infty}^0;X_1^\infty) \] is the excess entropy, measuring predictable structure shared between past and future.
Shannon-McMillan-Breiman theorem
Theorem. For a stationary ergodic finite-alphabet process, \[ -\frac1n\log P(X_1^n)\to\overline H(X) \quad\text{almost surely and in }L^1. \]
Proof outline. Express log likelihood as a sum of conditional log probabilities given finite pasts. Martingale convergence identifies their limit with the conditional law given the infinite past, and Birkhoff's ergodic theorem averages the stationary limiting information function. Uniform integrability gives \(L^1\) convergence.
Spectral formula for Gaussian processes
Theorem. A zero-mean stationary Gaussian process with spectral density \(f(\omega)>0\) has differential entropy rate \[ \overline h= \frac1{4\pi}\int_{-\pi}^{\pi} \log\{2\pi e\,f(\omega)\}\,\mathrm{d}\omega \] under standard integrability conditions.
This follows from Toeplitz determinant asymptotics applied to \(h(X_1^n)=\tfrac12\log\{(2\pi e)^n\det\Sigma_n\}\).