A prior on probability measures treats the sampling distribution itself as unknown. The central questions are support, conjugacy, and weak convergence.
Random probability measures
The parameter is now a probability measure \(P\), not a finite-dimensional vector.
Definition. A random probability measure is a measurable map \(P:\Omega\to\mathcal P(\mathcal X)\). A prior on \(\mathcal P(\mathcal X)\) induces a law for iid observations \(X_i\mid P\sim P\).
Definition. Weak neighborhoods of \(P_0\) are determined by finitely many bounded continuous test functions: \[ \left|\int f_j\,\mathrm{d}P-\int f_j\,\mathrm{d}P_0\right|<\varepsilon,\quad j=1,\ldots,k. \]
Weak support
A prior is useful only if it gives positive mass near plausible sampling laws.
Proposition. If every weak neighborhood of \(P_0\) has positive prior probability, then \(P_0\) belongs to the weak support of the prior.
Proof. This is the definition of support for a probability measure on a topological space: a point is in the support if all open neighborhoods of it have positive probability.
Posterior on measures
Bayes formula applies in the space of probability measures just as in parametric models.
Proposition. For iid observations and prior \(\Pi\) on \(\mathcal P(\mathcal X)\), the posterior is \[ \Pi(B\mid X_1,\ldots,X_n) =\frac{\int_B \prod_{i=1}^n p(X_i)\,\Pi(\mathrm{d}P)} {\int \prod_{i=1}^n p(X_i)\,\Pi(\mathrm{d}P)} \] when the measures have densities \(p\) relative to a common dominating measure.
Proof. This is ordinary Bayes formula with parameter \(P\). The likelihood is the product density of the iid sample under \(P\).
Discrete and continuous random measures
Some priors, such as the Dirichlet process, generate discrete probability measures almost surely even when the base measure is continuous.
Remark. Discreteness of the random measure does not prevent it from being useful for mixture modeling, because mixtures can smooth the induced sampling density.
Topologies on probability measures
Definition. On a Polish space \(\mathcal X\), weak convergence \[ P_n\Rightarrow P \] means \[ \int f\,\mathrm{d}P_n\to\int f\,\mathrm{d}P \quad\text{for every }f\in C_b(\mathcal X). \] The bounded-Lipschitz metric \[ d_{\mathrm{BL}}(P,Q) =\sup_{\|f\|_\infty+\operatorname{Lip}(f)\le1} \left|\int f\,\mathrm{d}(P-Q)\right| \] metrizes weak convergence.
Weak support is appropriate for distributional functionals continuous under weak convergence, but density estimation often needs stronger Hellinger, \(L^1\), or KL support after a smoothing kernel is applied.
Random probability measures as measurable maps
Definition. A random probability measure is a measurable map \[ P:\Omega\to\mathcal P(\mathcal X), \] where \(\mathcal P(\mathcal X)\) carries its Borel \(\sigma\)-field under weak topology. Equivalently, \(P(A)\) is measurable for every set in a countable convergence-determining class.
Proposition. The law of a random probability measure is determined by the joint laws of \[ \{P(A_1),\ldots,P(A_k)\} \] over finite measurable partitions \((A_1,\ldots,A_k)\) from a generating algebra, provided these finite-dimensional laws are projectively consistent.
Proof outline. Evaluation maps on a countable generating algebra generate the Borel \(\sigma\)-field of \(\mathcal P(\mathcal X)\). Consistent finite-dimensional laws define a probability law on the coordinate product by Kolmogorov extension. Additivity and total mass one, imposed on the finite partitions, restrict this law to probability-measure-valued coordinates.
Completely random measures and normalization
Definition. A completely random measure \(\mu\) has independent masses on disjoint measurable sets. If \[ 0<\mu(\mathcal X)<\infty \quad\text{almost surely}, \] then \[ P=\frac{\mu}{\mu(\mathcal X)} \] is a normalized random measure with dependent increments.
Example. A gamma process with base measure \(\alpha\) satisfies \[ \mu(A)\sim\operatorname{Gamma}(\alpha(A),1) \] independently over disjoint \(A\). Normalization gives a Dirichlet process \[ P\sim\operatorname{DP}(\alpha). \] The total mass \(\mu(\mathcal X)\) cancels from relative cell masses but controls neither the mean base distribution nor the resulting concentration parameter separately once \(\alpha\) is specified.
Moments of random measures
Definition. The mean measure of a random probability measure \(P\) is \[ \bar P(A)=E\{P(A)\}. \] For bounded \(f,g\), covariance is described by \[ \operatorname{Cov}\left( \int f\,\mathrm{d}P,\int g\,\mathrm{d}P \right). \]
Proposition. If \(X_1,X_2\mid P\stackrel{\mathrm{iid}}{\sim}P\), then \[ \operatorname{Cov}\{f(X_1),g(X_2)\} =\operatorname{Cov}\left( \int f\,\mathrm{d}P,\int g\,\mathrm{d}P \right). \]
Proof. Conditional independence gives \[ E\{f(X_1)g(X_2)\mid P\} =\left(\int f\,\mathrm{d}P\right) \left(\int g\,\mathrm{d}P\right). \] Take expectations and subtract the product of marginal means.
Randomness of \(P\) therefore induces positive or negative dependence among observations according to covariance of random integrals, even though observations are conditionally iid.
Weak support criterion
Proposition. Suppose a prior on \(\mathcal P(\mathcal X)\) can assign positive probability to arbitrary finite probability vectors over every finite measurable partition whose boundaries have \(P_0\)-mass zero. Then \(P_0\) belongs to its weak support.
Proof outline. A basic weak neighborhood of \(P_0\) is determined by finitely many bounded continuous integrals. Approximate each function by a simple function on a sufficiently fine partition with \(P_0\)-null boundaries. Closeness of cell masses then makes all integrals close. Positive prior mass for the cell-mass neighborhood gives positive mass to the weak neighborhood.
Discrete draws and smooth observations
If \[ P=\sum_{h\ge1}w_h\delta_{\theta_h} \] is discrete and observations are drawn directly from \(P\), ties occur with positive probability. In a mixture model \[ Y_i\mid\theta_i\sim K(\cdot\mid\theta_i), \qquad \theta_i\mid P\sim P, \] the induced sampling density \[ f_P(y)=\int K(y\mid\theta)P(\mathrm{d}\theta) =\sum_hw_hK(y\mid\theta_h) \] can be smooth. The random measure clusters latent parameters; the kernel determines observation smoothness.
Identifiability of random mixing laws
Remark. Posterior learning of \(P\) depends on identifiability of the mixture map \[ P\mapsto f_P. \] If two mixing measures generate the same sampling density, the data can identify only their equivalence class even when the prior lives on a rich space. Density contraction can be much faster and more stable than recovery of the mixing distribution itself.