The Gaussian linear model is the finite-dimensional projection prototype for regression. Generalized linear models retain a linear predictor while replacing Gaussian constant variance by an exponential-family mean-variance relation and a link function.
Linear projection and least squares
Definition. For a fixed \(n\times p\) design matrix \(X\) of rank \(p\), the linear model is \[ Y=X\beta+\varepsilon,\qquad \mathbb E(\varepsilon\mid X)=0,\qquad \operatorname{Cov}(\varepsilon\mid X)=\sigma^2I_n. \] The ordinary least-squares estimator minimizes \[ \|Y-Xb\|_2^2 \] over \(b\in\mathbb R^p\).
Normal equations and orthogonal projection. The unique minimizer is \[ \hat\beta=(X^\top X)^{-1}X^\top Y. \] The fitted vector and residual are \[ \hat Y=H Y,\qquad e=(I-H)Y,\qquad H=X(X^\top X)^{-1}X^\top, \] where \(H\) is the orthogonal projector onto \(\operatorname{col}(X)\). Consequently, \[ X^\top e=0,\qquad \|Y\|^2=\|\hat Y\|^2+\|e\|^2. \]
Proof. Differentiating the least-squares criterion gives \[ -2X^\top(Y-Xb)=0. \] Rank \(p\) makes \(X^\top X\) invertible and yields \(\hat\beta\). Direct calculation gives \(H^\top=H\) and \(H^2=H\), so \(H\) is an orthogonal projection. Since \(X^\top(I-H)=0\), residuals are orthogonal to the fitted subspace, and Pythagoras gives the norm decomposition.
Conditional moments. \[ \mathbb E(\hat\beta\mid X)=\beta,\qquad \operatorname{Cov}(\hat\beta\mid X) =\sigma^2(X^\top X)^{-1}. \]
Proof. The exact error identity \[ \hat\beta-\beta =(X^\top X)^{-1}X^\top\varepsilon \] gives both statements by conditional expectation and covariance.
Gauss-Markov optimality
Gauss-Markov theorem. Among all linear unbiased estimators \(AY\) of \(\beta\), OLS has minimum covariance: \[ \operatorname{Cov}(AY\mid X) -\operatorname{Cov}(\hat\beta\mid X) \succeq0. \]
Proof. Unbiasedness requires \(AX=I_p\). Write \[ A=(X^\top X)^{-1}X^\top+D, \qquad DX=0. \] The cross terms vanish because \(DX=0\), and \[ AA^\top =(X^\top X)^{-1}+DD^\top. \] Multiplication by \(\sigma^2\) gives the covariance decomposition.
Exact Gaussian inference
Normal linear model distributions. If \[ \varepsilon\mid X\sim N(0,\sigma^2I_n), \] then \[ \hat\beta\mid X\sim N\{\beta,\sigma^2(X^\top X)^{-1}\}, \] \[ \frac{\operatorname{RSS}}{\sigma^2} =\frac{\|e\|^2}{\sigma^2} \sim\chi_{n-p}^2, \] and \(\hat\beta\) is independent of \(\operatorname{RSS}\).
Proof. Both \(\hat\beta-\beta\) and \(e\) are linear transformations of a Gaussian vector. Their conditional covariance is \[ \sigma^2(X^\top X)^{-1}X^\top(I-H)=0, \] so they are independent. The residual projector \(I-H\) is symmetric idempotent of rank \(n-p\); diagonalizing it orthogonally shows that \(\varepsilon^\top(I-H)\varepsilon/\sigma^2\) is the sum of \(n-p\) independent squared standard normals.
Student inference for a contrast. Let \[ \hat\sigma^2=\frac{\operatorname{RSS}}{n-p}. \] For \(c\in\mathbb R^p\), \[ \frac{c^\top\hat\beta-c^\top\beta} {\hat\sigma\{c^\top(X^\top X)^{-1}c\}^{1/2}} \sim t_{n-p}. \]
Generalized least squares
If \(\operatorname{Cov}(\varepsilon\mid X)=\Sigma\) is known and positive definite, whitening gives \[ \hat\beta_{\mathrm{GLS}} =(X^\top\Sigma^{-1}X)^{-1} X^\top\Sigma^{-1}Y. \] It is the projection of \(Y\) onto \(\operatorname{col}(X)\) in the inner product \[ \langle u,v\rangle_{\Sigma^{-1}} =u^\top\Sigma^{-1}v. \] If \(\Sigma\) is unknown, feasible GLS must account for its estimation; misspecified GLS covariance is again a sandwich.
Exponential-family responses
Definition. A one-parameter exponential-dispersion family has density \[ p(y;\vartheta,\phi) =\exp\left\{ \frac{y\vartheta-b(\vartheta)} {a(\phi)}+c(y,\phi) \right\}. \] Its mean and variance are \[ \mu=b'(\vartheta),\qquad \operatorname{Var}(Y)=a(\phi)b''(\vartheta). \]
Proof. Normalization implies \[ \mathbb E\left[ \frac{Y-b'(\vartheta)}{a(\phi)} \right]=0, \] giving the mean. Differentiating this score identity once more gives the variance relation.
Definition (GLM). A generalized linear model specifies independent responses with \[ \mathbb E(Y_i\mid x_i)=\mu_i,\qquad \operatorname{Var}(Y_i\mid x_i) =a(\phi)V(\mu_i), \] and a link \[ g(\mu_i)=\eta_i=x_i^\top\beta. \] The link is canonical when \(\eta_i=\vartheta_i\).
Score and information
GLM score and Fisher information. Let \[ d_i=\frac{\mathrm{d}\mu_i}{\mathrm{d}\eta_i}, \qquad w_i=\frac{d_i^2}{a(\phi)V(\mu_i)}. \] Then \[ U(\beta) =X^\top D V^{-1}(Y-\mu)/a(\phi) =X^\top W D^{-1}(Y-\mu), \] with the first expression interpreted componentwise, and \[ I(\beta)=X^\top W X. \] For a canonical link, \[ U(\beta)=\frac1{a(\phi)}X^\top(Y-\mu), \qquad W_{ii}=\frac{V(\mu_i)}{a(\phi)}. \]
Proof. For one response, \[ \frac{\partial\ell_i}{\partial\beta} =\frac{Y_i-\mu_i}{a(\phi)} \frac{\mathrm{d}\vartheta_i}{\mathrm{d}\mu_i} \frac{\mathrm{d}\mu_i}{\mathrm{d}\eta_i}x_i. \] Since \(\mathrm{d}\mu/\mathrm{d}\vartheta=b''(\vartheta)=V(\mu)\), this becomes \[ \frac{Y_i-\mu_i}{a(\phi)V(\mu_i)}d_ix_i. \] Summing gives the score. Its covariance uses \(\operatorname{Var}(Y_i)=a(\phi)V(\mu_i)\), producing \(X^\top W X\).
Fisher scoring and IRLS
Iteratively reweighted least squares. At a current value \(\beta^{(k)}\), define \[ z_i^{(k)} =\eta_i^{(k)} +(Y_i-\mu_i^{(k)}) \left.\frac{\mathrm{d}\eta_i} {\mathrm{d}\mu_i}\right|_{\mu_i^{(k)}}. \] The Fisher-scoring update is \[ \beta^{(k+1)} =\{X^\top W^{(k)}X\}^{-1} X^\top W^{(k)}z^{(k)}. \]
Proof. Fisher scoring solves \[ I(\beta^{(k)}) (\beta^{(k+1)}-\beta^{(k)}) =U(\beta^{(k)}). \] From the score formula, \[ U=X^\top W(z-X\beta^{(k)}). \] Substitute and rearrange to obtain the weighted least-squares update.
Canonical examples
Examples.
- Gaussian identity: \(b(\eta)=\eta^2/2\), \(\mu=\eta\), and constant weights.
- Bernoulli logistic: \(b(\eta)=\log(1+e^\eta)\), \(\mu=e^\eta/(1+e^\eta)\), and \(w=\mu(1-\mu)\).
- Poisson log-linear: \(b(\eta)=e^\eta\), \(\mu=e^\eta\), and \(w=\mu\).
Fixed-dimensional asymptotic normality
GLM MLE limit. Suppose \(p\) is fixed, the true means remain in a regular interior region, \[ \frac1nX^\top W(\beta_0)X\to I_0 \] with \(I_0\) positive definite, a Lindeberg condition holds for score contributions, and the MLE is consistent. Then \[ \sqrt n(\hat\beta-\beta_0) \rightsquigarrow N(0,I_0^{-1}). \]
Proof. The score is a sum of independent centered vectors and satisfies \[ n^{-1/2}U(\beta_0)\rightsquigarrow N(0,I_0). \] Taylor expansion gives \[ 0=U(\hat\beta) =U(\beta_0)-I_n(\tilde\beta) (\hat\beta-\beta_0), \] where \(n^{-1}I_n(\tilde\beta)\to_pI_0\). Solve and apply Slutsky.
Separation, dispersion, and model failure
In logistic regression, complete separation gives a direction \(v\) along which fitted probabilities approach the observed labels and the likelihood supremum occurs at \(\|\beta\|=\infty\). Existence must therefore be checked before reporting a Hessian-based standard error. Penalized or bias-reduced estimators change the target and finite-sample procedure.
For Poisson regression, conditional variance exceeding the mean produces overdispersion. A quasi-likelihood can retain the mean model while estimating a dispersion factor, or a negative-binomial model can specify an explicit variance function. If only the conditional mean is trusted, sandwich covariance provides inference for the estimating-equation target. High-dimensional GLMs additionally require sparsity, restricted curvature, score concentration, and debiasing; fixed-\(p\) likelihood theory does not survive simply by replacing \(p\) with \(p_n\).