Regression models combine likelihood geometry with prior regularization. In linear Gaussian models everything is explicit; in generalized linear models the same ideas appear through curvature and approximation.
Gaussian linear regression
Conjugate Gaussian regression is the algebraic model behind ridge-like Bayesian regularization.
Definition. Let \(Y\mid\beta,\sigma^2\sim N(X\beta,\sigma^2 I_n)\) and \(\beta\sim N(b_0,V_0)\), with \(\sigma^2\) known. The posterior precision and mean are \[ V_n^{-1}=V_0^{-1}+\sigma^{-2}X^\top X,\qquad b_n=V_n(V_0^{-1}b_0+\sigma^{-2}X^\top Y). \]
Proof. The log posterior is the sum of two quadratic forms: \[ -\frac{1}{2}(\beta-b_0)^\top V_0^{-1}(\beta-b_0) -\frac{1}{2\sigma^2}\|Y-X\beta\|^2. \] Collecting the quadratic and linear terms in \(\beta\) gives the displayed normal posterior.
Ridge interpretation
The posterior mode solves a penalized least-squares problem.
Proposition. If \(b_0=0\) and \(V_0=\tau^2 I\), then the posterior mode minimizes \[ \|Y-X\beta\|^2+\frac{\sigma^2}{\tau^2}\|\beta\|^2. \]
Proof. Multiplying the negative log posterior by \(2\sigma^2\) removes constants and yields the squared-error term plus the quadratic prior penalty.
Generalized linear models
For GLMs, the likelihood is no longer quadratic, but local curvature still drives posterior concentration.
Definition. In a canonical GLM, \[ p_\beta(y_i)=\exp\{y_i x_i^\top\beta-b(x_i^\top\beta)+c(y_i)\}. \] The log posterior is \(\ell_n(\beta)+\log\pi(\beta)\).
Remark. Laplace approximation uses the Hessian of \(-\ell_n(\beta)-\log\pi(\beta)\) at the posterior mode. This connects Bayesian GLMs to iteratively reweighted least squares and asymptotic normality.
Separation and prior regularization
In logistic regression, separation can destroy the maximum likelihood estimator but a proper prior can still yield a proper posterior.
Proposition. A proper prior with sufficiently light tails can regularize separated logistic regression by making the posterior normalizing integral finite.
Proof. The likelihood is bounded above by \(1\). Posterior propriety therefore follows if the prior is proper. The prior tail then determines whether posterior moments are finite.
Gaussian linear model in matrix form
Theorem. Let \[ y\mid\beta,\sigma^2\sim N_n(X\beta,\sigma^2I_n), \qquad \beta\mid\sigma^2\sim N_p(b_0,\sigma^2V_0). \] Conditional on \(\sigma^2\), \[ \beta\mid y,\sigma^2 \sim N_p(b_n,\sigma^2V_n), \] where \[ V_n=(X^\top X+V_0^{-1})^{-1}, \qquad b_n=V_n(X^\top y+V_0^{-1}b_0). \]
Proof. The negative log posterior kernel is \[ \frac1{2\sigma^2} \left[ (y-X\beta)^\top(y-X\beta) +(\beta-b_0)^\top V_0^{-1}(\beta-b_0) \right]. \] Collecting quadratic and linear terms in \(\beta\) gives precision \(X^\top X+V_0^{-1}\) and mean satisfying the stated normal equations.
Corollary (ridge posterior mean). If \(b_0=0\) and \(V_0=\lambda^{-1}I_p\), then \[ E(\beta\mid y,\sigma^2) =(X^\top X+\lambda I_p)^{-1}X^\top y, \] the ridge estimator. The Bayesian model additionally supplies a posterior covariance and predictive distribution; the equality of point estimates does not make cross-validated ridge and a fixed-prior Bayesian analysis identical.
Posterior prediction in linear regression
Proposition. With known \(\sigma^2\), for a future response at design vector \(x_\star\), \[ y_\star\mid y \sim N\left( x_\star^\top b_n,\, \sigma^2\{1+x_\star^\top V_nx_\star\} \right). \]
Proof. Conditional on \(\beta\), \(y_\star=x_\star^\top\beta+\varepsilon_\star\), where \(\varepsilon_\star\sim N(0,\sigma^2)\) is independent of the posterior \(\beta\). Means add, and independent variances give \[ \sigma^2x_\star^\top V_nx_\star+\sigma^2. \]
Generalized linear model posterior
Definition. For a canonical exponential-family GLM, \[ p(y_i\mid\eta_i,\phi) =\exp\left[ \frac{y_i\eta_i-b(\eta_i)}{a(\phi)} +c(y_i,\phi) \right], \qquad \eta_i=x_i^\top\beta. \] With prior density \(\pi(\beta)\), the log posterior is \[ \log\pi(\beta\mid y) =\sum_i\frac{y_ix_i^\top\beta-b(x_i^\top\beta)} {a(\phi)} +\log\pi(\beta)+C. \]
Proposition. If \(b\) is convex and \(-\log\pi(\beta)\) is convex, then the negative log posterior is convex. If either likelihood curvature on the design span or prior curvature is strictly positive in every direction, the posterior mode is unique.
Proof. The likelihood Hessian is \[ \frac1{a(\phi)} \sum_i b''(x_i^\top\beta)x_ix_i^\top \succeq0. \] Add the positive semidefinite Hessian of \(-\log\pi\). Strict positive definiteness gives strict convexity and uniqueness.
P?lya-Gamma augmentation for logistic regression
Theorem (augmentation identity). For \(\psi\in\mathbb R\), \(a\in\mathbb R\), and \(b>0\), \[ \frac{e^{a\psi}}{(1+e^\psi)^b} =2^{-b}e^{(a-b/2)\psi} \int_0^\infty e^{-\omega\psi^2/2} p_{\mathrm{PG}}(\omega\mid b,0) \,\mathrm{d}\omega. \] Thus, after introducing \(\omega_i\mid\beta\sim\mathrm{PG}(1,x_i^\top\beta)\), a Gaussian prior yields a Gaussian full conditional for \(\beta\).
Proof outline. Rewrite \[ \frac{e^{a\psi}}{(1+e^\psi)^b} =2^{-b}e^{(a-b/2)\psi} \cosh^{-b}(\psi/2). \] The defining Laplace transform of a P?lya-Gamma variable is \[ E(e^{-\omega\psi^2/2}) =\cosh^{-b}(\psi/2). \] Conditional on \(\omega_i\), the log likelihood is quadratic in each \(x_i^\top\beta\), giving normal conjugacy.
Separation and tail behavior
Proposition. If logistic data are completely separated by a vector \(v\), then along \(\beta=cv\) the likelihood increases to a positive finite limit as \(c\to\infty\). Under a proper Gaussian prior the posterior is proper and has all moments; under a proper Cauchy prior it is proper, but some posterior means can fail to exist when a predictor is a solitary separator.
Proof. Separation makes every fitted probability approach its observed binary outcome, so the likelihood approaches its supremum without finite maximizer. Multiplication by any proper prior remains integrable because the likelihood is bounded. Gaussian tails make every polynomial moment integrable. Cauchy tails decay only quadratically, and in a separation direction the likelihood supplies no exponential tail decay, so first moments may diverge in configurations where that direction enters a coefficient without cancellation.
Prior scale should therefore be chosen after standardizing continuous predictors and considering plausible changes on the response scale. Posterior regularization is a modeling claim about effect sizes, not merely a numerical fix for optimization.