VC dimension measures combinatorial regularity. Smoothness, compactness, and shape restrictions provide a different source of small entropy: nearby domain points constrain nearby function values. The exponent in the entropy rate records how many local degrees of freedom remain after those analytic constraints are imposed.
Parametric Lipschitz families
Proposition. Let \(\Theta\subset\mathbb R^q\) have diameter at most \(D\), and suppose \[ |f_\theta(x)-f_\eta(x)| \leq L(x)\|\theta-\eta\|_2. \] Then, for every \(Q\) with \(\|L\|_{Q,2}<\infty\), \[ N\!\left( \varepsilon\|L\|_{Q,2}, \{f_\theta:\theta\in\Theta\},L_2(Q) \right) \leq\left(1+\frac{2D}{\varepsilon}\right)^q. \]
Proof. An \(\varepsilon\)-net \(\theta_1,\ldots,\theta_N\) of \(\Theta\) induces a net \(f_{\theta_1},\ldots,f_{\theta_N}\), because \[ \|f_\theta-f_{\theta_j}\|_{Q,2} \leq\|L\|_{Q,2}\|\theta-\theta_j\|_2. \] Apply the Euclidean volume bound from the preceding page. \(\square\)
Thus a \(q\)-parameter model has logarithmic entropy \[ \log N(\varepsilon)\lesssim q\log(1/\varepsilon). \] This is qualitatively smaller than the polynomial entropy of an infinite-dimensional smoothness ball.
Hölder balls
Definition. Write \(s=m+\alpha\), where \(m\in\mathbb N_0\) and \(0<\alpha\leq1\). The Hölder norm on \([0,1]^d\) is \[ \|f\|_{C^s} = \max_{|\beta|\leq m}\|D^\beta f\|_\infty + \max_{|\beta|=m} \sup_{x\ne y} \frac{|D^\beta f(x)-D^\beta f(y)|}{\|x-y\|_2^\alpha}. \] Let \(C_M^s([0,1]^d)=\{f:\|f\|_{C^s}\leq M\}\).
Theorem (Kolmogorov–Tikhomirov entropy). For fixed \(d,s,M\), constants \(0<c<C<\infty\) exist such that, for sufficiently small \(\varepsilon\), \[ c\varepsilon^{-d/s} \leq \log N(\varepsilon,C_M^s([0,1]^d),\|\cdot\|_\infty) \leq C\varepsilon^{-d/s}. \]
Proof outline of the upper bound. Partition the cube into cells of side length \(h\asymp\varepsilon^{1/s}\). On each cell, Taylor's theorem approximates \(f\) by its degree-\(m\) jet at one grid point with error \(O(h^s)=O(\varepsilon)\). Quantize the finitely many derivative values in increments matched to their Taylor coefficients. Hölder compatibility means that, after fixing one cell, only a bounded number of quantized choices is possible across each neighboring face. The logarithm of the total number of compatible codes is therefore proportional to the number of cells, \(h^{-d}\asymp\varepsilon^{-d/s}\), rather than that number times \(\log(1/\varepsilon)\). For the lower bound, place disjoint \(s\)-smooth bumps of height \(\varepsilon\) on a grid of spacing \(h\), and use binary choices of their signs. The resulting \(2^{c h^{-d}}\) functions remain pairwise separated in sup norm. \(\square\)
Corollary (Donsker threshold). Since \(L_2(P)\)-distance is bounded by sup-norm distance, \[ \int_0^1 \sqrt{\log N(\varepsilon,C_M^s,L_2(P))} \,\mathrm{d}\varepsilon \lesssim \int_0^1\varepsilon^{-d/(2s)}\,\mathrm{d}\varepsilon. \] The integral is finite exactly when \(s>d/2\). Subject to measurability, a uniformly bounded Hölder ball is therefore universally Donsker when \(s>d/2\).
The threshold is not merely decorative. Below it, the class contains too many small-scale oscillations for a supremum-norm Gaussian limit under arbitrary \(P\). It is the same smoothness-versus-dimension balance that appears in nonparametric rates.
Sobolev balls and compact embeddings
Theorem. On a bounded regular domain \(\Omega\subset\mathbb R^d\), the unit ball of \(W^{s,2}(\Omega)\) satisfies \[ \log N(\varepsilon,B_{W^{s,2}},L_2(\Omega)) \asymp\varepsilon^{-d/s}. \] In particular, the embedding \(W^{s,2}(\Omega)\hookrightarrow L_2(\Omega)\) is compact for \(s>0\).
One way to see the rate is spectral. Expanding in eigenfunctions of an elliptic operator turns the Sobolev ball into an ellipsoid with semiaxes \(a_j\asymp j^{-s/d}\). Covering the first \(m\asymp\varepsilon^{-d/s}\) coordinates and bounding the remaining tail yields the upper rate; packing sign vectors in the first \(m\) suitably scaled coordinates yields the lower rate.
Shape-constrained classes
Theorem (monotone functions). For the class \(\mathcal M\) of nondecreasing functions \([0,1]\to[0,1]\), \[ \log N_{[]}(\varepsilon,\mathcal M,L_2(P)) \lesssim\varepsilon^{-1} \] uniformly over probability laws \(P\). Hence \(\mathcal M\) is \(P\)-Donsker for every \(P\).
A monotone function may jump and need not be smooth, but it can spend its total variation only once. The entropy exponent \(1\) records this order restriction. Convexity likewise restricts how slopes may change. For uniformly bounded convex functions on a compact convex subset of \(\mathbb R^d\), the characteristic interior sup-norm entropy rate is \[ \log N(\varepsilon)\asymp\varepsilon^{-d/2}, \] with boundary behavior requiring care when no Lipschitz bound is imposed.
Sequence ellipsoids
Example. In \(\ell^2\), let \[ \mathcal E(a)= \left\{\theta:\sum_{j\geq1}\frac{\theta_j^2}{a_j^2}\leq1\right\}, \qquad a_1\geq a_2\geq\cdots\downarrow0. \] Coordinates with \(a_j\ll\varepsilon\) are invisible at resolution \(\varepsilon\). A useful entropy approximation is \[ \log N(\varepsilon,\mathcal E(a),\ell^2) \asymp \sum_{j:a_j\gtrsim\varepsilon} \log\!\left(\frac{a_j}{\varepsilon}\right). \] If \(a_j\asymp j^{-s/d}\), this sum has order \(\varepsilon^{-d/s}\), recovering the Sobolev rate.
Entropy exponents and statistical regimes
If a local class satisfies \[ \log N(\varepsilon)\asymp\varepsilon^{-\alpha}, \] then its Dudley integral near zero behaves as \[ \int_0^\delta\varepsilon^{-\alpha/2}\,\mathrm{d}\varepsilon. \] It is finite at zero when \(\alpha<2\), logarithmically critical at \(\alpha=2\), and divergent when \(\alpha>2\). Divergence of this particular integral does not prove that no useful limit or rate exists; it says that global Donsker control by this entropy route is too coarse. Localization, a weaker topology, or problem-specific cancellation may still succeed.