Numerical analysis begins by separating the mathematical problem, the algorithm used to approximate it, and the several notions of error by which the approximation is judged.
Numerical problem
Definition. A numerical problem is a map \(\Phi:X\to Y\) between normed spaces, together with a class of admissible data \(x\in X\). An algorithm replaces \(\Phi(x)\) by a computable object \(A(x)\), often after discretizing \(X\), \(Y\), or both.
Forward error and residual
Definition. The forward error is \(\|A(x)-\Phi(x)\|_Y\). If the target is characterized by an equation \(F(y,x)=0\), the residual is \(\|F(A(x),x)\|\). A small residual is meaningful only when the equation is well conditioned.
Backward error
Definition. The backward error of an approximation \(\widehat y\) is the smallest perturbation of the data for which \(\widehat y\) is exact: \(\inf\{\|\Delta x\|: \Phi(x+\Delta x)=\widehat y\}\). Backward error is often the cleanest way to express algorithmic stability.
Error decomposition
Proposition. In a finite computation the total error is usually decomposed as approximation error plus propagated rounding error. For a discretization parameter \(h\) and floating-point unit roundoff \(u\), one often obtains a bound of the form \(C_1h^p+C_2u h^{-q}\), which explains why simply making \(h\) smaller can eventually make the answer worse.
Proof by comparison
Proof. A typical numerical proof compares three objects: the exact continuum solution, the exact discrete solution, and the computed discrete solution. The first gap is controlled by approximation theory; the second by stability of the discrete algorithm.
Problems, data, and algorithms
A numerical problem is a map \(F:\mathcal X\to\mathcal Y\) between normed data and solution spaces. An algorithm implemented in finite arithmetic produces \(\widehat y=A(x)\), which may differ from \(F(x)\) through discretization, iteration truncation, measurement error, and rounding. These sources should be separated before an error bound is interpreted.
Definition. The absolute and relative forward errors are \[ \|\widehat y-F(x)\|, \qquad \frac{\|\widehat y-F(x)\|}{\|F(x)\|}, \] when \(F(x)\ne0\). A backward error is \[ \eta(x,\widehat y) =\inf\{\|\delta x\|:\widehat y=F(x+\delta x)\}, \] with a relative version obtained by dividing by \(\|x\|\).
Forward error asks whether the answer is close. Backward error asks whether the computed answer is exact for nearby data. The latter is often easier to prove because it follows the algebra performed by the algorithm.
Well-posedness and conditioning
Definition. A problem is well posed at \(x\) if a solution exists, is locally unique, and depends continuously on the data. If \(F\) is Fréchet differentiable, its absolute local condition number is \[ \kappa_{\mathrm{abs}}(x)=\|\mathrm DF(x)\|, \] and its normwise relative condition number is \[ \kappa_{\mathrm{rel}}(x) =\frac{\|x\|}{\|F(x)\|}\|\mathrm DF(x)\|. \]
Proposition (first-order error principle). If \(\widehat y=F(x+\delta x)\), \(F\) is differentiable at \(x\), and \(\delta x\to0\), then \[ \frac{\|\widehat y-F(x)\|}{\|F(x)\|} \le \kappa_{\mathrm{rel}}(x) \frac{\|\delta x\|}{\|x\|} +o\!\left(\frac{\|\delta x\|}{\|x\|}\right). \]
Proof. Fréchet differentiability gives \[ F(x+\delta x)-F(x) =\mathrm DF(x)[\delta x]+o(\|\delta x\|). \] Take norms and insert \(\|\delta x\|=(\|\delta x\|/\|x\|)\|x\|\).
Residuals and error estimators
If a problem is written \(G(y)=0\), the residual is \(r=G(\widehat y)\). A small residual implies a small solution error only when the inverse problem is conditioned. If \(\mathrm DG(y^*)\) is invertible, \[ \widehat y-y^* =\mathrm DG(y^*)^{-1}r+o(\|r\|). \] For \(Ax=b\), \(r=b-A\widehat x\) and \[ \frac{\|\widehat x-x\|}{\|x\|} \le\kappa(A)\frac{\|r\|}{\|b\|} \] under compatible norms. A tiny residual can coexist with a large error when \(A\) is ill conditioned.
A priori and a posteriori bounds
An a priori estimate depends on regularity and algorithm parameters before computation, such as \(\|u-u_h\|\le Ch^p\|u\|_{H^{p+1}}\). An a posteriori estimate uses the computed object, typically residuals and local indicators. Adaptive methods need a posteriori estimates that are both reliable \(\|e\|\le C_{\mathrm rel}\eta\) and efficient \(\eta\le C_{\mathrm eff}(\|e\|+\text{oscillation})\).
Cost and accuracy
Complexity should state an accuracy target and a computational model. Dense Gaussian elimination costs \(O(n^3)\) operations but only \(O(n^2)\) storage; an iterative method may cost \(O(\operatorname{nnz}(A))\) per step but require a condition-dependent number of steps. Bit complexity also counts precision, which may need to grow as conditioning worsens. Operation counts without stability and accuracy are incomplete comparisons.