The empirical measure is a random probability measure, while the empirical process is a centered random linear functional. Its covariance is determined entirely by \(P\), and finite classes reduce exactly to multivariate probability. Infinite-class theory begins when the class size grows beyond a single union bound.
Random signed functionals
Definition. For a measurable \(f\) with \(P|f|\lt\infty\), \[ P_nf=\frac1n\sum_{i=1}^nf(X_i),\qquad \mathbb G_nf=\frac1{\sqrt n}\sum_{i=1}^n\{f(X_i)-Pf\}. \] On a vector space of functions, both \(P_n-P\) and \(\mathbb G_n\) are linear: \[ \mathbb G_n(af+bg) =a\mathbb G_nf+b\mathbb G_ng. \]
The empirical measure \(P_n\) is positive and has total mass one. The centered measure \(P_n-P\) is signed and has total mass zero. This is why constants vanish: \[ \mathbb G_n(f+c)=\mathbb G_nf. \]
Mean and covariance
Empirical-process covariance. If \(f,g\in L^2(P)\), then \[ \mathbb E(\mathbb G_nf)=0 \] and, for every \(n\), \[ \operatorname{Cov}(\mathbb G_nf,\mathbb G_ng) =P(fg)-Pf\,Pg. \]
Proof. Centering gives the zero mean. Independence makes cross-observation covariances vanish, so \[ \begin{aligned} \operatorname{Cov}(\mathbb G_nf,\mathbb G_ng) &=\frac1n\sum_{i=1}^n \operatorname{Cov}\{f(X_i),g(X_i)\}\\ &=P(fg)-Pf\,Pg. \end{aligned} \]
Increment variance. \[ \mathbb E\{\mathbb G_n(f-g)\}^2 =\rho_P(f,g)^2, \] where \[ \rho_P(f,g)^2 =P\{(f-g)-P(f-g)\}^2. \]
Finite-dimensional central limit theorem
Theorem. For \(f_1,\ldots,f_k\in L^2(P)\), \[ (\mathbb G_nf_1,\ldots,\mathbb G_nf_k) \rightsquigarrow N_k(0,\Sigma), \] where \[ \Sigma_{ij}=P(f_if_j)-Pf_i\,Pf_j. \]
Proof. For every \(a\in\mathbb R^k\), \[ \sum_{j=1}^ka_j\mathbb G_nf_j =\frac1{\sqrt n}\sum_{i=1}^n \left[ \sum_{j=1}^ka_jf_j(X_i) -P\sum_{j=1}^ka_jf_j \right]. \] The scalar central limit theorem gives a centered normal limit with variance \(a^\top\Sigma a\). Cramer-Wold proves the vector convergence.
Finite-class uniform laws
Finite-class law of large numbers. If \(\mathcal F=\{f_1,\ldots,f_N\}\) is finite and every \(f_j\in L^1(P)\), then \[ \|P_n-P\|_{\mathcal F}\to0 \quad\text{almost surely}. \]
Proof. The strong law gives \((P_n-P)f_j\to0\) almost surely for each \(j\). The intersection of finitely many probability-one events still has probability one, and the maximum of finitely many convergent absolute values tends to zero.
Sub-Gaussian bound for a finite bounded class
Finite-class maximal inequality. Suppose \(\mathcal F\) has \(N\) elements and every centered coordinate satisfies \[ \mathbb E\exp\{\lambda\mathbb G_nf\} \le\exp(\lambda^2\sigma^2/2) \quad\text{for all }\lambda\in\mathbb R. \] Then \[ \mathbb E\max_{f\in\mathcal F} |\mathbb G_nf| \le\sigma\sqrt{2\log(2N)}. \]
Proof. Apply the exponential maximum bound to the \(2N\) variables \(\{\mathbb G_nf,-\mathbb G_nf:f\in\mathcal F\}\): \[ \begin{aligned} \mathbb E\max_f|\mathbb G_nf| &\le\frac1\lambda \log\sum_{f,\pm} \mathbb E e^{\pm\lambda\mathbb G_nf}\\ &\le\frac{\log(2N)}{\lambda} +\frac{\lambda\sigma^2}{2}. \end{aligned} \] Minimize at \(\lambda=\sqrt{2\log(2N)}/\sigma\).
Bounded coordinates. If every \(f\in\mathcal F\) takes values in an interval of length \(b\), Hoeffding's lemma gives the preceding condition with \(\sigma=b/2\), and hence \[ \mathbb E\|\mathbb G_n\|_{\mathcal F} \le\frac b2\sqrt{2\log(2N)}. \]
Envelopes and normalization
Definition. A measurable function \(F\ge0\) is an envelope for \(\mathcal F\) if \[ |f(x)|\le F(x) \quad\text{for every }f\in\mathcal F,\ x\in\mathcal X. \] Moment assumptions are stated through \(PF^r\), and entropy is often normalized by \(\|F\|_{Q,2}\).
An integrable envelope alone does not control an arbitrary infinite class: the class can contain all measurable indicators and overfit the sample. Complexity measures quantify how many genuinely different coordinates exist after identifying functions close in \(L^2(Q)\).
Centered and uncentered symmetrized processes
Independent Rademacher signs \(\varepsilon_i\) lead to \[ \mathbb G_n^\circ f =\frac1{\sqrt n}\sum_{i=1}^n \varepsilon_if(X_i). \] Conditionally on the data, this process is centered and has increment metric \[ d_{P_n}(f,g) =\{P_n(f-g)^2\}^{1/2}. \] Symmetrization compares \(\mathbb G_n\) to \(\mathbb G_n^\circ\); chaining then controls the latter through random covering numbers in this empirical metric.