Modern inference often follows adaptive modeling, high-dimensional regularization, or prediction. The classical principles remain: define the target under a probability law, state the conditioning and uniformity class, isolate a leading pivot or exchangeability argument, and account for every data-dependent choice that affects its law.
Familywise error and false discovery rate
Definition. For \(m\) hypotheses, let \(V\) be the number of rejected true nulls and \(R\) the total number rejected. The familywise error rate and false discovery rate are \[ \operatorname{FWER}=P(V\ge1), \qquad \operatorname{FDR} =\mathbb E\left(\frac{V}{R\vee1}\right). \]
Bonferroni control. If every true-null p-value is valid, rejecting \(H_j\) when \(p_j\le\alpha/m\) gives \[ \operatorname{FWER}\le\alpha \] under arbitrary dependence.
Proof. If \(\mathcal H_0\) is the true-null set, the union bound and validity give \[ P(V\ge1) \le\sum_{j\in\mathcal H_0} P(p_j\le\alpha/m) \le|\mathcal H_0|\alpha/m\le\alpha. \]
Benjamini-Hochberg procedure
FDR control under independence. Order the p-values \[ p_{(1)}\le\cdots\le p_{(m)} \] and let \[ \hat k=\max\{k:p_{(k)}\le qk/m\}, \] with \(\hat k=0\) if the set is empty. Reject \(p_j\le p_{(\hat k)}\). If true-null p-values are independent uniform variables and independent of false-null p-values, then \[ \operatorname{FDR}=\frac{m_0}{m}q\le q. \]
Proof. For a true null \(j\), let \(R^{(-j)}\) be the number of BH rejections after replacing \(p_j\) by zero. On the event that \(j\) is rejected, the step-up property gives \(R=R^{(-j)}\), and rejection is equivalent to \[ p_j\le qR^{(-j)}/m. \] Since \(R^{(-j)}\) depends only on the other p-values, \[ \begin{aligned} \mathbb E\frac{1_{\{j\text{ rejected}\}}}{R\vee1} &= \mathbb E\left[ \frac{1_{\{p_j\le qR^{(-j)}/m\}}} {R^{(-j)}}\right]\\ &=\mathbb E\left[ \frac{qR^{(-j)}/m}{R^{(-j)}}\right] =\frac qm, \end{aligned} \] with zero contribution when \(R^{(-j)}=0\). Summing over the \(m_0\) true nulls gives \(m_0q/m\).
Positive dependence permits related proofs; arbitrary dependence requires more conservative thresholds, such as the Benjamini-Yekutieli harmonic correction. Data-dependent filtering must be independent or incorporated into the validity argument.
Sample splitting and selective targets
If one subsample selects a model and an independent subsample performs classical inference conditional on the selected model, the second-stage pivot retains its nominal law for the random selected target. This avoids selection bias at the cost of information. Repeated splitting can stabilize results but creates another aggregation problem.
Conditional validity by independent splitting. Let \(\widehat M\) be measurable with respect to training data \(\mathcal D_1\), and let \(C_{\widehat M}(\mathcal D_2)\) have conditional coverage \[ P\{\theta_{\widehat M}\in C_{\widehat M}(\mathcal D_2) \mid\mathcal D_1\}\ge1-\alpha \] for independent \(\mathcal D_2\). Then its unconditional coverage is at least \(1-\alpha\).
Proof. Take expectations of the conditional coverage inequality with respect to \(\mathcal D_1\).
Selective inference instead conditions on a selection event, such as a polyhedron produced by the lasso, and derives the truncated conditional law of a statistic. The target is then conditional on or indexed by selection. Ignoring this target change can make a mathematically valid selective interval answer a different scientific question.
Debiased high-dimensional estimation
In the linear model \(Y=X\beta_0+\varepsilon\) with \(p\gg n\), the lasso is biased by its \(\ell^1\) penalty. Let \(\hat\beta\) be a lasso estimate and \(\hat\Theta\) approximate the inverse Gram matrix. The debiased estimator is \[ \tilde\beta =\hat\beta+ \hat\Theta\frac{X^\top(Y-X\hat\beta)}n. \] Substitution of \(Y=X\beta_0+\varepsilon\) gives the exact decomposition \[ \tilde\beta-\beta_0 =\hat\Theta\frac{X^\top\varepsilon}n +(I-\hat\Theta\hat\Sigma) (\hat\beta-\beta_0), \qquad \hat\Sigma=\frac{X^\top X}{n}. \]
Remainder bound. \[ \|(I-\hat\Theta\hat\Sigma) (\hat\beta-\beta_0)\|_\infty \le \|I-\hat\Theta\hat\Sigma\|_{\max} \|\hat\beta-\beta_0\|_1. \] Thus asymptotic normality of a coordinate follows when the product is \(o_p(n^{-1/2})\) and the leading weighted score satisfies a central limit theorem.
Proof. For each row \(j\), \[ |e_j^\top(I-\hat\Theta\hat\Sigma) (\hat\beta-\beta_0)| \le \max_k|(I-\hat\Theta\hat\Sigma)_{jk}| \sum_k|\hat\beta_k-\beta_{0k}|. \] Take the maximum over \(j\).
Sparsity, restricted eigenvalues, precision-matrix approximation, and tail conditions make this product small. Debiasing is not automatic after fitting any high-dimensional predictor.
Orthogonal machine-learning scores
For a low-dimensional parameter \(\theta_0\) identified by \[ P\psi(W;\theta_0,\eta_0)=0, \] Neyman orthogonality makes the derivative with respect to \(\eta\) vanish at the truth. With cross-fitting, \[ \sqrt n(\hat\theta-\theta_0) =-A^{-1}\frac1{\sqrt n}\sum_{i=1}^n \psi(W_i;\theta_0,\eta_0)+o_p(1) \] provided nuisance-product remainders are \(o_p(n^{-1/2})\). Flexible prediction algorithms enter only through verifiable convergence and positivity conditions.
Conformal prediction
Split-conformal coverage. Fit a prediction rule on training data. On an independent calibration sample of size \(n\), compute exchangeable nonconformity scores \(R_1,\ldots,R_n\). Let \[ \hat q=R_{(k)},\qquad k=\lceil(n+1)(1-\alpha)\rceil, \] with \(\hat q=\infty\) if \(k=n+1\). For a new covariate \(X_{n+1}\), include responses \(y\) whose score \(R(X_{n+1},y)\le\hat q\). If the calibration and test observations are exchangeable, then \[ P\{Y_{n+1}\in C(X_{n+1})\} \ge1-\alpha. \]
Proof. Conditional on the fitted training rule, the \(n+1\) scores \(R_1,\ldots,R_n,R_{n+1}\) are exchangeable. With randomized ties, the rank of \(R_{n+1}\) is uniform on \(\{1,\ldots,n+1\}\); conservatively, its probability of exceeding the \(k\)-th calibration order statistic is at most \[ \frac{n+1-k}{n+1}\le\alpha. \]
This is marginal coverage over a fresh exchangeable observation. Exact conditional coverage given \(X=x\) for every distribution is generally impossible with finite-width sets without structural assumptions. Covariate shift, time dependence, or adaptive feedback require weighted or sequential conformal arguments with new assumptions.
Randomization and knockoffs
Model-X knockoffs construct synthetic covariates \(\tilde X\) satisfying pairwise exchangeability with \(X\) while being conditionally independent of \(Y\) given \(X\). Antisymmetric feature statistics then give signs that are fair coins for null variables, enabling an FDR threshold. Validity depends on the covariate distribution or a correct estimate of it; ordinary variable importance scores do not automatically inherit knockoff guarantees.
What modern validity must state
Every adaptive procedure should identify:
- the estimand or predictive target after selection;
- the probability statement, including conditioning and whether it is pointwise or uniform;
- the independence, exchangeability, sparsity, nuisance-rate, or design assumptions;
- the leading stochastic expansion and the remainder that must be controlled;
- the effect of tuning, splitting, and repeated use of the data.
Multiple testing, debiasing, orthogonal scores, selective inference, and conformal prediction are developed further from prediction and complexity viewpoints in statistical learning.