Gaussian process priors put Gaussian laws on function spaces. Their geometry is governed by covariance kernels, reproducing-kernel Hilbert spaces, and small-ball probabilities.
Gaussian processes
A Gaussian process is determined by its mean and covariance kernel.
Definition. A random function \(W\) on \(\mathcal X\) is a Gaussian process with mean \(m\) and covariance \(K\) if every finite vector \((W(x_1),\ldots,W(x_k))\) is multivariate normal with mean \(m(x_j)\) and covariance \(K(x_i,x_j)\).
Proposition. A symmetric positive semidefinite kernel \(K\) defines consistent finite-dimensional Gaussian distributions.
Proof. Positive semidefiniteness gives valid covariance matrices on every finite set. Consistency under marginalization follows from the corresponding property of multivariate normal distributions, and Kolmogorov extension gives a stochastic process.
RKHS and support
The RKHS is not the typical sample-path space, but it controls shifts and support.
Definition. The reproducing-kernel Hilbert space \(\mathbb H\) associated with \(K\) is the Hilbert closure of finite linear combinations of \(K(x,\cdot)\), with inner product satisfying \(f(x)=\langle f,K(x,\cdot)\rangle_{\mathbb H}\).
Theorem. For many Gaussian measures on Banach spaces, the support is the closure of the RKHS in the Banach norm.
Proof. The Cameron-Martin space gives directions along which Gaussian measure can be shifted absolutely continuously. Small balls around RKHS elements have positive probability, and closure gives all support points; points outside the closure have separating neighborhoods of zero Gaussian mass.
Concentration function
Posterior contraction for Gaussian priors is often expressed through approximation plus small-ball terms.
Definition. For truth \(f_0\), the concentration function is \[ \varphi_{f_0}(\varepsilon) =\inf_{\|h-f_0\|\le\varepsilon,h\in\mathbb H}\frac12\|h\|_{\mathbb H}^2 -\log \mathbb P(\|W\|\le\varepsilon). \]
Remark. The first term measures RKHS approximation cost; the second measures how much prior mass a small ball receives.
Kernel scale and adaptation
Length-scale hyperparameters change both smoothness and concentration.
Remark. Hierarchical Gaussian process priors place a prior on scale parameters so that the posterior can adapt to unknown regularity, but adaptation requires enough hyperprior mass near the scale that balances bias and variance.
Gaussian measures and covariance operators
Definition. A process \(W=(W_t)_{t\in T}\) is Gaussian if every finite vector \((W_{t_1},\ldots,W_{t_k})\) is multivariate normal. It is determined by \[ m(t)=EW_t,\qquad K(s,t)=\operatorname{Cov}(W_s,W_t), \] where \(K\) is symmetric and positive semidefinite: \[ \sum_{i,j}a_ia_jK(t_i,t_j)\ge0. \]
Theorem (Karhunen-Loève representation). Let \(K\) be a continuous covariance kernel on a compact domain with eigenpairs \((\lambda_j,\phi_j)\) in \(L^2\). Then \[ W(t)=m(t)+\sum_{j\ge1} \sqrt{\lambda_j}Z_j\phi_j(t), \qquad Z_j\stackrel{\mathrm{iid}}{\sim}N(0,1), \] with convergence in \(L^2(\Omega\times T)\).
Proof outline. The covariance operator is compact, self-adjoint, and positive, so the spectral theorem supplies eigenpairs. Coefficients \[ Z_j=\lambda_j^{-1/2} \int(W-m)\phi_j \] are centered jointly Gaussian with identity covariance and hence independent. Parseval's identity makes the mean-square remainder equal the tail sum of eigenvalues, which vanishes.
Reproducing kernel Hilbert space
Definition. The RKHS \(\mathbb H_K\) is the completion of finite linear combinations of \[ K_t(\cdot)=K(t,\cdot) \] under \[ \left\langle \sum_ia_iK_{t_i}, \sum_jb_jK_{s_j} \right\rangle_{\mathbb H} =\sum_{i,j}a_ib_jK(t_i,s_j). \] It satisfies the reproducing identity \[ h(t)=\langle h,K_t\rangle_{\mathbb H}. \]
Proposition. In the Karhunen-Loève basis, \[ \mathbb H_K =\left\{ h=\sum_jh_j\phi_j: \sum_j\frac{h_j^2}{\lambda_j}<\infty \right\}, \] with norm squared equal to the displayed sum.
Proof. The covariance operator maps \(\phi_j\) to \(\lambda_j\phi_j\). Its square-root range consists of coefficients \(h_j=\sqrt{\lambda_j}a_j\) for \((a_j)\in\ell^2\), giving norm \(\sum a_j^2=\sum h_j^2/\lambda_j\).
Cameron-Martin shift
Theorem. Let \(\mathcal L(W)\) be a centered Gaussian measure on a separable Banach space. The shifted law of \(W+h\) is equivalent to the law of \(W\) if \(h\in\mathbb H_K\), and mutually singular if \(h\notin\mathbb H_K\). For \(h\in\mathbb H_K\), the Radon-Nikodym derivative has the formal form \[ \frac{\mathrm{d}\mathcal L(W+h)} {\mathrm{d}\mathcal L(W)}(W) =\exp\left\{ \langle W,h\rangle_{\mathbb H} -\frac12\|h\|_{\mathbb H}^2 \right\}, \] where the first term is interpreted as the associated Gaussian linear functional.
The RKHS therefore measures the exponential cost of translating the prior toward an approximating function. Typical GP draws usually lie outside their RKHS, just as Brownian paths are almost surely nondifferentiable while its RKHS consists of absolutely continuous paths with square-integrable derivative.
Support theorem
Theorem. The support of a centered Gaussian measure in a separable Banach space \(\mathbb B\) is the \(\mathbb B\)-closure of its RKHS: \[ \operatorname{supp}\mathcal L(W) =\overline{\mathbb H_K}^{\,\mathbb B}. \]
Proof outline. Cameron-Martin equivalence implies that every neighborhood of an RKHS element has positive probability whenever a neighborhood of zero does. Closure extends this inclusion. Conversely, a point outside the closed RKHS can be separated by a continuous linear functional that vanishes on the RKHS. That functional has zero Gaussian variance and hence is almost surely zero under the prior, excluding a neighborhood of the point.
Concentration function
Definition. For truth \(f_0\in\mathbb B\), \[ \varphi_{f_0}(\varepsilon) =\inf_{\substack{h\in\mathbb H_K\\ \|h-f_0\|_{\mathbb B}<\varepsilon}} \frac12\|h\|_{\mathbb H_K}^2 -\log P(\|W\|_{\mathbb B}<\varepsilon). \] The first term is approximation cost; the second is centered small-ball cost.
Theorem (GP contraction template). In a regular Gaussian or regression experiment, if \[ \varphi_{f_0}(\varepsilon_n) \lesssim n\varepsilon_n^2, \] and suitable sieve entropy and tail conditions hold, then the posterior contracts at rate \(\varepsilon_n\) in the experiment's natural metric.
Conjugate GP regression
Theorem. Let \[ f\sim\operatorname{GP}(m,K),\qquad y_i=f(x_i)+\varepsilon_i,\quad \varepsilon_i\sim N(0,\sigma^2). \] Write \(K_X=(K(x_i,x_j))\), \(k_\star=(K(x_\star,x_i))\), and \(m_X=(m(x_i))\). Then \[ f(x_\star)\mid y \sim N\left( m(x_\star)+ k_\star^\top(K_X+\sigma^2I)^{-1}(y-m_X), \right. \] \[ \left. K(x_\star,x_\star) -k_\star^\top(K_X+\sigma^2I)^{-1}k_\star \right). \]
Proof. The joint vector \((f(x_\star),y)\) is Gaussian with covariance blocks \[ \begin{bmatrix} K(x_\star,x_\star)&k_\star^\top\\ k_\star&K_X+\sigma^2I \end{bmatrix}. \] Apply the conditional multivariate-normal formula.
Kernel regularity and rescaling
Matérn kernels have polynomial spectral tails and finite smoothness controlled by their order; squared-exponential kernels have exponentially decaying spectra and analytic RKHS functions. A fixed overly smooth prior can still estimate rough truths, but may do so at a suboptimal rate because RKHS approximation is expensive. A rescaled process \[ W^a(t)=W(at) \] changes effective length scale and RKHS geometry. A hyperprior on \(a\) can adapt when it allocates enough mass near the oracle scale while controlling extreme scales. Posterior learning of a length scale is a concentration problem, not merely maximizing a covariance marginal likelihood.