Inverse problems try to recover causes from observed effects. They are often ill posed, so numerical analysis must include regularization from the start.
Ill-posed inverse problem
Definition. An inverse problem is ill posed when existence, uniqueness, or continuous dependence on data fails. Compact forward operators often have singular values tending to zero, which amplifies noise in inverse reconstruction.
Tikhonov regularization
Definition. Tikhonov regularization replaces \(Ax\approx b\) by minimizing \(\|Ax-b\|^2+\lambda\|Lx\|^2\). The parameter \(\lambda\) balances data fit against smoothness, size, or another structural penalty.
Spectral filtering
Proposition. In the SVD basis, regularization multiplies each noisy inverse coefficient by a filter factor. Truncated SVD discards small singular values; Tikhonov shrinks them continuously.
Bias-variance tradeoff
Theorem. Regularization introduces bias to control noise amplification. Under source conditions and noise bounds, an appropriate choice of \(\lambda\) yields convergence rates determined by smoothness and singular-value decay.
Statistical link
Remark. Many inverse problems are also statistical estimation problems. Numerical regularization, Bayesian priors, penalized likelihood, and minimax rates often describe the same stabilization phenomenon in different languages.
Compact operators and ill-posedness
Let \(K:X\to Y\) be compact between Hilbert spaces, with singular system \[ Kv_j=\sigma_ju_j,\qquad K^*u_j=\sigma_jv_j,\qquad \sigma_j\downarrow0. \] The formal inverse \[ x=\sum_j\frac{\langle y,u_j\rangle}{\sigma_j}v_j \] amplifies high-index noise. The range of an infinite-rank compact operator is not closed, so inversion is discontinuous.
Tikhonov regularization
Theorem. The minimizer of \[ \|Kx-y^\delta\|^2+\alpha\|x\|^2 \] is \[ x_\alpha^\delta =(K^*K+\alpha I)^{-1}K^*y^\delta =\sum_j \frac{\sigma_j}{\sigma_j^2+\alpha} \langle y^\delta,u_j\rangle v_j. \]
Proof. The strictly convex objective has normal equation \((K^*K+\alpha I)x=K^*y^\delta\). Expanding in the singular basis gives the filter factors.
The factor \(\sigma/(\sigma^2+\alpha)\) behaves like \(1/\sigma\) for well-resolved modes and suppresses modes with \(\sigma^2\ll\alpha\).
Bias--noise balance
If \(\|y^\delta-y\|\le\delta\), then \[ \|x_\alpha^\delta-x^\dagger\| \le \underbrace{\|x_\alpha-x^\dagger\|}_{\text{regularization bias}} + \underbrace{\frac{\delta}{2\sqrt\alpha}}_{\text{noise amplification}}. \] The noise bound follows from \(\sup_{\sigma\ge0}\sigma/(\sigma^2+\alpha)=1/(2\sqrt\alpha)\). Convergence requires \(\alpha\downarrow0\) and \(\delta/\sqrt\alpha\to0\).
Source conditions and rates
If \[ x^\dagger=(K^*K)^\nu w, \] then the bias is controlled by a power of \(\alpha\). Balancing it with the noise term yields an algebraic convergence rate determined by smoothness \(\nu\). These source conditions link recoverability to alignment with singular vectors, not merely ordinary differentiability.
Parameter choice
The discrepancy principle chooses \(\alpha\) so \[ \|Kx_\alpha^\delta-y^\delta\|\approx\tau\delta,\qquad\tau>1. \] Cross-validation, generalized cross-validation, L-curves, and empirical Bayes are alternatives with different assumptions. A regularized solution without a parameter-selection rule is not a complete numerical procedure.
Iterative regularization
Landweber iteration \[ x_{k+1}=x_k+\omega K^*(y^\delta-Kx_k), \qquad 0<\omega<2/\|K\|^2, \] acts as a spectral filter whose effective cutoff moves outward with \(k\). Early stopping is the regularization parameter. Iterating to full convergence fits noise and recovers the unstable least-squares inverse.