Hierarchical Bayes introduces latent structure and shared hyperparameters. Exchangeability explains why such models are natural for repeated units that have no intrinsic ordering.
Conditional independence and partial pooling
A hierarchical model separates unit-level variation from population-level variation.
Definition. A basic two-level model has \[ Y_i\mid\theta_i\sim P_{\theta_i},\qquad \theta_i\mid\eta\stackrel{\mathrm{iid}}{\sim}G_\eta,\qquad \eta\sim H. \] Conditional on \(\eta\), the unit parameters are independent; marginally, they are dependent.
Proposition. The posterior distribution of \(\theta_i\) uses information from both \(Y_i\) and the other observations through the hyperparameter \(\eta\).
Proof. The posterior factorizes as \[ \pi(\eta,\theta_{1:n}\mid y)\propto h(\eta)\prod_{i=1}^n g_\eta(\theta_i)p_{\theta_i}(y_i). \] Integrating out \(\eta\) couples the conditional posterior of each \(\theta_i\) to all data \(y_{1:n}\).
Normal means shrinkage
The normal hierarchical model gives the clearest formula for partial pooling.
Example. If \(Y_i\mid\theta_i\sim N(\theta_i,\sigma^2)\) and \(\theta_i\mid\mu,\tau^2\sim N(\mu,\tau^2)\), then for fixed \((\mu,\tau^2)\), \[ \mathbb E(\theta_i\mid Y_i,\mu,\tau^2) =\frac{\tau^2}{\tau^2+\sigma^2}Y_i+\frac{\sigma^2}{\tau^2+\sigma^2}\mu. \] The posterior mean shrinks \(Y_i\) toward the population mean \(\mu\).
Exchangeability and de Finetti representation
Exchangeability turns symmetry into a mixture representation.
Theorem. For an infinite exchangeable sequence \(Y_1,Y_2,\ldots\) taking values in a standard Borel space, there exists a random probability measure \(P\) such that, conditionally on \(P\), the \(Y_i\) are iid with common law \(P\).
Proof. A full proof is a measure-theoretic theorem. The key idea is that empirical measures form a reverse martingale under the exchangeable sigma-field, and conditional distributions given that tail information become iid directing laws.
Hyperpriors and identifiability
Hyperpriors must be chosen with the same care as ordinary priors.
Remark. Weak data information about variance components can make posterior behavior sensitive to hyperprior tails near zero or infinity. In hierarchical models, prior predictive checking is especially important because hyperparameters control the scale of entire groups.
Normal-normal partial pooling
Theorem. Suppose \[ Y_j\mid\theta_j\sim N(\theta_j,\sigma_j^2), \qquad \theta_j\mid\mu,\tau^2\sim N(\mu,\tau^2), \] with \(\mu,\tau\) fixed. Then \[ \theta_j\mid Y_j,\mu,\tau \sim N(m_j,v_j), \] where \[ v_j=\left(\frac1{\sigma_j^2}+\frac1{\tau^2}\right)^{-1}, \qquad m_j=B_j\mu+(1-B_j)Y_j, \qquad B_j=\frac{\sigma_j^2}{\sigma_j^2+\tau^2}. \]
Proof. Completing the square in \[ \frac{(Y_j-\theta_j)^2}{\sigma_j^2} +\frac{(\theta_j-\mu)^2}{\tau^2} \] gives posterior precision \(\sigma_j^{-2}+\tau^{-2}\) and precision-weighted mean. Algebra converts that mean into the displayed convex combination.
Groups with noisy observations have larger \(B_j\) and shrink more toward \(\mu\). When \(\tau\to0\), complete pooling results; when \(\tau\to\infty\), the estimates approach no pooling.
Example. If \(\sigma_j=4\), \(\tau=2\), \(\mu=10\), and \(Y_j=20\), then \[ B_j=\frac{16}{20}=0.8,\qquad m_j=0.8(10)+0.2(20)=12. \] The extreme observation is heavily shrunk because its sampling variance is four times the between-group variance.
Exchangeability and de Finetti mixtures
Theorem (de Finetti for Bernoulli sequences). An infinite Bernoulli sequence \((X_i)\) is exchangeable if and only if there is a probability measure \(\Pi\) on \([0,1]\) such that \[ P(X_1=x_1,\ldots,X_n=x_n) =\int_0^1 \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i}\,\Pi(\mathrm{d}p) \] for every \(n\).
Proof outline. The reverse direction follows because a mixture of iid laws is invariant under permutations. For the forward direction, exchangeability makes the empirical proportions \(S_n/n\) a reverse-martingale sequence relative to the tail of symmetric \(\sigma\)-fields, so they converge almost surely to a random \(P\). Conditional mixed moments of \(P\) reproduce all finite exchangeable probabilities, yielding conditional iid Bernoulli sampling given \(P\).
Finite exchangeability. A finitely exchangeable vector need not extend to an infinite exchangeable sequence and need not be an exact iid mixture. The hierarchical representation is therefore a substantive extendibility assumption, not merely a synonym for permutation symmetry in one finite dataset.
Learning the population distribution
With unknown \((\mu,\tau)\), each group informs both its own \(\theta_j\) and the population distribution. The marginal model \[ Y_j\mid\mu,\tau \sim N(\mu,\sigma_j^2+\tau^2) \] shows that \(\tau\) is learned from between-group dispersion after removing sampling variance. With very few groups, this separation is weak and the prior on \(\tau\) materially affects shrinkage.
Proposition. If all \(\sigma_j=\sigma\) and \(\mu\) is known, the marginal log-likelihood for \(v=\tau^2\) is \[ \ell(v) =-\frac J2\log(\sigma^2+v) -\frac{\sum_j(Y_j-\mu)^2}{2(\sigma^2+v)} +C. \] Its unconstrained maximizer satisfies \[ \sigma^2+\widehat v =\frac1J\sum_j(Y_j-\mu)^2, \] with \(\widehat v\) truncated at zero.
Proof. Differentiate: \[ \ell'(v) =-\frac{J}{2(\sigma^2+v)} +\frac{\sum_j(Y_j-\mu)^2}{2(\sigma^2+v)^2}. \] Set this to zero and enforce \(v\ge0\).
Centered and noncentered parameterizations
Definition. The centered form samples \[ \theta_j\sim N(\mu,\tau^2). \] The noncentered form introduces \[ \eta_j\sim N(0,1),\qquad \theta_j=\mu+\tau\eta_j. \]
They define the same probability law but different posterior geometry. When group data are weak, centered \(\theta_j\) and small \(\tau\) create a narrow funnel, whereas \(\eta_j\) remains order one; noncentering often improves Hamiltonian sampling. With highly informative group data, centering can be more efficient. Partial noncentering interpolates between the two.
Posterior prediction for a new group
Proposition. A new latent effect has posterior predictive law \[ p(\theta_{\mathrm{new}}\mid y) =\int N(\theta_{\mathrm{new}}\mid\mu,\tau^2) \Pi(\mathrm{d}\mu,\mathrm{d}\tau\mid y). \] Its variance decomposes as \[ \operatorname{Var}(\theta_{\mathrm{new}}\mid y) =E(\tau^2\mid y) +\operatorname{Var}(\mu\mid y). \]
Proof. Apply total variance conditional on \((\mu,\tau)\). Conditional mean is \(\mu\), and conditional variance is \(\tau^2\).
Prediction for a new group is generally more uncertain than estimation of the population mean. Confusing these targets is a common source of intervals that are much too narrow.