Bayesian inference starts from a joint probability law on parameters and observations. The posterior is not an informal update rule; it is a regular conditional distribution obtained by disintegrating that joint law.
Bayesian experiments and joint laws
The model, the prior, and the observation law should be treated as one probability space. This viewpoint keeps the distinction between the sampling distribution \(P_\theta\), the prior \(\Pi\), and the posterior kernel precise.
Definition. Let \((\mathcal X,\mathcal A)\) be the sample space and \((\Theta,\mathcal T)\) the parameter space. A Bayesian experiment consists of a Markov kernel \(\theta\mapsto P_\theta\) from \(\Theta\) to \(\mathcal X\) and a prior probability \(\Pi\) on \(\Theta\). The joint law is \[ M(\mathrm{d}\theta,\mathrm{d}x)=P_\theta(\mathrm{d}x)\Pi(\mathrm{d}\theta). \] The marginal law of the data is \(P_\Pi(A)=\int P_\theta(A)\,\Pi(\mathrm{d}\theta)\).
Proposition. If \(\Theta\) and \(\mathcal X\) are standard Borel spaces, then there exists a posterior kernel \(x\mapsto\Pi(\cdot\mid x)\) satisfying \[ M(B\times A)=\int_A \Pi(B\mid x)\,P_\Pi(\mathrm{d}x) \] for all \(B\in\mathcal T\) and \(A\in\mathcal A\).
Proof. This is the regular conditional probability theorem for probability measures on standard Borel spaces. Applying it to the coordinate projection \((\theta,x)\mapsto x\) disintegrates the joint law \(M\) with respect to its \(x\)-marginal \(P_\Pi\). The resulting conditional law of the first coordinate is exactly the posterior kernel.
Bayes formula in dominated models
Most calculations use a dominated version of the preceding disintegration. The likelihood appears only after choosing a common dominating measure.
Theorem. Suppose \(P_\theta\ll\mu\) with density \(p_\theta\), and define \[ m(x)=\int_\Theta p_\theta(x)\,\Pi(\mathrm{d}\theta). \] For \(P_\Pi\)-almost every \(x\) with \(m(x)>0\), \[ \Pi(B\mid x)=\frac{\int_B p_\theta(x)\,\Pi(\mathrm{d}\theta)}{m(x)}. \]
Proof. For rectangles \(B\times A\), \[ M(B\times A)=\int_B\int_A p_\theta(x)\,\mu(\mathrm{d}x)\Pi(\mathrm{d}\theta) =\int_A\int_B p_\theta(x)\Pi(\mathrm{d}\theta)\,\mu(\mathrm{d}x). \] Also \(P_\Pi(\mathrm{d}x)=m(x)\mu(\mathrm{d}x)\). The displayed ratio gives a probability kernel and reproduces the joint law on rectangles, hence by a monotone class argument it is a version of the posterior.
Support and identifiability
The prior is part of the statistical model. No amount of data can give posterior mass to a region that has prior mass zero.
Proposition. If \(\Pi(B)=0\), then \(\Pi(B\mid X)=0\) for \(P_\Pi\)-almost every \(X\). If two parameters induce the same sampling law, \(P_{\theta_1}=P_{\theta_2}\), then the likelihood cannot distinguish them.
Proof. The first claim follows from Bayes formula, or directly from \(M(B\times\mathcal X)=0\). For the second claim, \(p_{\theta_1}(X)/p_{\theta_2}(X)=1\) \(P_{\theta_1}\)-almost surely under domination, so posterior odds between the two points are changed only by prior odds and not by data.
Likelihood ratios and posterior odds
Posterior odds are the cleanest finite-dimensional expression of Bayesian updating.
Example. For two simple hypotheses \(\theta_0,\theta_1\), \[ \frac{\Pi(\theta_1\mid X)}{\Pi(\theta_0\mid X)} =\frac{\Pi(\theta_1)}{\Pi(\theta_0)} \frac{p_{\theta_1}(X)}{p_{\theta_0}(X)}. \] Thus the posterior odds equal prior odds times the likelihood ratio.
Posterior kernels beyond densities
Theorem (posterior disintegration). Let \(\Theta\) and \(\mathcal X\) be standard Borel spaces, let \(\Pi\) be a prior on \(\Theta\), and let \(P_\theta\) be a probability kernel from \(\Theta\) to \(\mathcal X\). Under the joint law \[ Q(\mathrm{d}\theta,\mathrm{d}x) =\Pi(\mathrm{d}\theta)P_\theta(\mathrm{d}x), \] there exists a posterior kernel \(x\mapsto\Pi(\cdot\mid x)\), unique for the marginal law \(m\) of \(X\), such that \[ Q(A\times B) =\int_B\Pi(A\mid x)\,m(\mathrm{d}x). \]
Proof. This is the existence theorem for regular conditional distributions on standard Borel spaces. Apply it to the coordinate map \(\Theta\) conditional on \(X\). Uniqueness follows because two versions have equal integrals over every measurable \(B\), hence agree \(m\)-almost surely on a countable generating class and then on the full \(\sigma\)-field by a monotone-class argument.
This formulation is essential in function-space models, where no Lebesgue density exists. Bayes' formula is then a Radon-Nikodym identity rather than a quotient of elementary densities.
Theorem (dominated Bayes formula). Suppose \(P_\theta\ll\mu\) with jointly measurable density \(p_\theta(x)\), and define \[ m(x)=\int_\Theta p_\vartheta(x)\,\Pi(\mathrm{d}\vartheta). \] For \(0<m(x)<\infty\), \[ \Pi(A\mid x) =\frac{\int_Ap_\theta(x)\,\Pi(\mathrm{d}\theta)} {\int_\Theta p_\theta(x)\,\Pi(\mathrm{d}\theta)}. \]
Proof. For measurable \(A,B\), \[ Q(A\times B) =\int_B\int_Ap_\theta(x)\,\Pi(\mathrm{d}\theta)\mu(\mathrm{d}x) =\int_B \frac{\int_Ap_\theta(x)\Pi(\mathrm{d}\theta)}{m(x)} m(x)\mu(\mathrm{d}x). \] Since \(m(x)\mu(\mathrm{d}x)\) is the marginal law of \(X\), the quotient satisfies the defining disintegration identity.
Sequential coherence
Proposition. If \(X_1,\ldots,X_n\) are conditionally independent given \(\theta\), updating first with \(x_{1:k}\) and then with \(x_{k+1:n}\) gives the same posterior as one-step updating: \[ \Pi(\mathrm{d}\theta\mid x_{1:n}) \propto \left\{\prod_{i=k+1}^np_\theta(x_i)\right\} \Pi(\mathrm{d}\theta\mid x_{1:k}). \]
Proof. The right-hand side is proportional to \[ \left\{\prod_{i=k+1}^np_\theta(x_i)\right\} \left\{\prod_{i=1}^kp_\theta(x_i)\right\}\Pi(\mathrm{d}\theta), \] which is the full posterior kernel. The omitted normalizing constants do not depend on \(\theta\).
A complete beta-binomial calculation
Example. Let \[ X\mid\theta\sim\operatorname{Binomial}(n,\theta), \qquad \theta\sim\operatorname{Beta}(a,b). \] After observing \(X=x\), \[ \theta\mid x\sim\operatorname{Beta}(a+x,b+n-x). \] Indeed, the posterior kernel is \[ \theta^{x}(1-\theta)^{n-x} \theta^{a-1}(1-\theta)^{b-1} =\theta^{a+x-1}(1-\theta)^{b+n-x-1}. \] Consequently, \[ E(\theta\mid x) =\frac{a+x}{a+b+n} =\frac{a+b}{a+b+n}\frac{a}{a+b} +\frac{n}{a+b+n}\frac{x}{n}, \] an explicit precision-weighted compromise between prior and empirical proportions. Its posterior variance is \[ \frac{(a+x)(b+n-x)} {(a+b+n)^2(a+b+n+1)}. \]
Identifiability and posterior learning
Proposition. If \(P_\theta=P_{\theta'}\) for distinct \(\theta,\theta'\), no amount of data can make their posterior odds depart from their prior odds: \[ \frac{\pi(\theta\mid X^{(n)})} {\pi(\theta'\mid X^{(n)})} =\frac{\pi(\theta)}{\pi(\theta')} \quad P_\theta^{(n)}\text{-almost surely} \] for discrete prior support.
Proof. Equal sampling laws give likelihood ratio one for every sample outside a common null set. The posterior-odds identity then leaves only prior odds.
Prior support is necessary for learning the truth, but it is not sufficient for a useful posterior rate. The likelihood must separate neighborhoods, and the prior must place enough local mass near the data-generating distribution. These requirements become the testing and concentration conditions used later in the note.