Mixture priors model densities by averaging kernels against an unknown mixing distribution. They are central because they turn discrete random measures into smooth density priors.
Kernel mixtures
A mixture density is a linear smoothing of a mixing distribution.
Definition. Given kernel \(k(x\mid z)\) and mixing distribution \(G\), the mixture density is \[ p_G(x)=\int k(x\mid z)\,G(\mathrm{d}z). \]
Proposition. If \(G_n\Rightarrow G\) weakly and \(z\mapsto k(x\mid z)\) is bounded continuous for each \(x\), then \(p_{G_n}(x)\to p_G(x)\) pointwise.
Proof. This is the definition of weak convergence applied to the bounded continuous test function \(z\mapsto k(x\mid z)\).
Dirichlet process mixtures
A discrete random mixing measure can still induce a continuous random density.
Definition. A Dirichlet process mixture uses \(G\sim\operatorname{DP}(\alpha G_0)\) and density \(p_G\). Even though \(G\) is discrete almost surely, \(p_G\) is smooth when \(k\) is smooth.
Remark. The mixture map \(G\mapsto p_G\) regularizes the random measure. This is why Dirichlet process mixtures are useful for density estimation.
KL support for mixtures
Consistency requires prior mass near the true density in KL neighborhoods.
Proposition. If the kernel family can approximate \(p_0\) in KL divergence and the prior on \(G\) assigns positive mass to weak neighborhoods of approximating mixing laws, then the induced density prior has KL support at \(p_0\).
Proof. Choose a mixing law \(G^*\) with \(K(p_0,p_{G^*})<\varepsilon\). Continuity of the mixture map in the relevant topology gives a neighborhood of \(G^*\) whose image remains in a KL neighborhood. Positive prior mass of that neighborhood gives KL support.
Rates and kernel bandwidth
Contraction rates depend on approximation by mixtures and on the complexity of the mixing distribution.
Remark. Location-scale normal mixtures can adapt to smooth densities, but the rate calculation must control tails, bandwidth, entropy of mixture classes, and prior mass near approximating mixtures.
Kernel-mixture map
Definition. For a kernel density \(K_\sigma(x-\mu)\) and mixing measure \(G\), \[ f_{G,\sigma}(x) =\int K_\sigma(x-\mu)\,G(\mathrm{d}\mu). \] A location-scale mixture additionally integrates over \(\sigma\): \[ f_G(x) =\int K_\sigma(x-\mu)\, G(\mathrm{d}\mu,\mathrm{d}\sigma). \]
Proposition. If \(K_\sigma\) is bounded and continuous, then weak convergence \(G_n\Rightarrow G\) implies pointwise convergence \[ f_{G_n,\sigma}(x)\to f_{G,\sigma}(x) \] for every \(x\). Under a common integrable envelope, convergence also holds in \(L^1\).
Proof. For fixed \(x\), the map \(\mu\mapsto K_\sigma(x-\mu)\) is bounded and continuous, so weak convergence gives the integral limit. An integrable envelope and pointwise convergence permit dominated convergence in \(x\).
Approximation by Gaussian convolutions
Theorem. If \(f_0\in L^1(\mathbb R)\) is a density and \(\varphi_\sigma\) is a centered Gaussian density of variance \(\sigma^2\), then \[ \|f_0*\varphi_\sigma-f_0\|_1\to0 \quad\text{as }\sigma\downarrow0. \] If \(f_0\) has \(\beta\)-order smoothness in an appropriate Hölder or Sobolev class and a bias-corrected approximation is used when necessary, the error is typically \[ \|f_0*\varphi_\sigma-f_0\| \lesssim\sigma^\beta. \]
Proof. Gaussian kernels form an approximate identity. Write \[ f_0*\varphi_\sigma-f_0 =\int\{f_0(\cdot-u)-f_0(\cdot)\} \varphi_\sigma(u)\,\mathrm{d}u. \] Translation continuity in \(L^1\), concentration of \(\varphi_\sigma\) near zero, and dominated convergence prove the first claim. Taylor or Fourier bounds give the smoothness rate.
Finite mixture approximation
Proposition. On a compact mixing region, a continuous Gaussian convolution can be approximated uniformly by a finite mixture with \[ N\asymp\sigma^{-1} \] regularly spaced support points in one dimension, up to logarithmic refinement for tail and KL control.
Proof outline. Partition the mixing region into cells of diameter \(O(\sigma)\) and replace mixing mass in each cell by an atom at a representative point. Lipschitz control of the Gaussian kernel in its location parameter bounds the integrated discretization error. Truncate mixing tails separately.
Kullback-Leibler support of mixture priors
Theorem (support template). Suppose:
- \(f_0\) can be approximated in KL divergence by some finite kernel mixture \(f_{G^\star,\sigma^\star}\);
- the prior on \(G\) assigns positive weak mass near \(G^\star\);
- the scale prior assigns positive mass near \(\sigma^\star\);
- kernel and tail conditions make the mixture map KL-continuous locally.
Then \[ \Pi\{K(f_0,f_{G,\sigma})<\varepsilon\}>0 \] for every \(\varepsilon>0\).
Proof. Choose an approximating finite mixture within \(\varepsilon/2\). Local weak and scale neighborhoods have positive prior probability. Continuity makes every mixture in a sufficiently small product neighborhood lie within the remaining KL budget. The product prior gives positive mass to that neighborhood.
Rate balance for smooth densities
For a \(\beta\)-smooth one-dimensional density, bandwidth \(\sigma\) gives squared approximation error roughly \[ \sigma^{2\beta}. \] The effective number of mixture components is roughly \(\sigma^{-1}\), so stochastic complexity is of order \[ \frac{\sigma^{-1}\log n}{n}. \] Balancing yields \[ \sigma_n\asymp (n/\log n)^{-1/(2\beta+1)}, \qquad \varepsilon_n\asymp (n/\log n)^{-\beta/(2\beta+1)}, \] up to model-specific logarithms. Adaptive scale priors must put enough mass near \(\sigma_n\) while penalizing scales that make sieve entropy too large.
Density and mixing-distribution metrics
Remark. Small Hellinger distance between \(f_G\) and \(f_{G_0}\) need not imply fast Wasserstein recovery of \(G\). Deconvolution through a smooth kernel is ill-posed, and finite-mixture singularities make component parameters converge at nonstandard rates when components merge. Every contraction theorem must name whether it concerns the density, mixing measure, number of components, or predictive distribution.
Label symmetry
Proposition. In a finite \(K\)-component mixture with exchangeable component priors, the posterior is invariant under all \(K!\) label permutations.
Proof. Both likelihood \[ \prod_i\sum_{k=1}^Kw_kK(y_i\mid\theta_k) \] and prior are unchanged by simultaneously permuting weights and component parameters. Bayes' formula preserves the symmetry.
Label switching is therefore not an MCMC defect. Component-specific summaries require an identification convention or permutation-invariant functionals. Density and predictive summaries are naturally invariant.
Posterior computation
Blocked stick-breaking truncation, slice sampling, marginal Chinese-restaurant samplers, split-merge moves, and sequential Monte Carlo expose different aspects of mixture geometry. A large occupied-cluster autocorrelation can coexist with stable density estimates, while apparently stable component labels can conceal mode trapping. Diagnostics should target the invariant quantity used in inference.