Here Newton's method is treated as a solver for nonlinear equations, not as an optimization method for minimizing an objective.
Newton iteration
Definition. For \(F:\mathbb R^d\to\mathbb R^d\), Newton's method solves \(J_F(x_k)s_k=-F(x_k)\) and updates \(x_{k+1}=x_k+s_k\), where \(J_F\) is the Jacobian.
Local quadratic convergence
Theorem. If \(F(x^\star)=0\), \(J_F(x^\star)\) is nonsingular, and \(J_F\) is locally Lipschitz, then Newton iterates started sufficiently close to \(x^\star\) converge quadratically.
Kantorovich theorem
Theorem. Kantorovich-type results give computable conditions under which Newton iterates exist, remain in a controlled ball, and converge. They combine an inverse-Jacobian bound with a Lipschitz bound on the Jacobian.
Damping and line search
Definition. A damped Newton method uses \(x_{k+1}=x_k+\alpha_k s_k\), with \(0\lt\alpha_k\le1\), chosen to reduce a merit function such as \(\|F(x)\|^2\). Damping trades local speed for global reliability.
Relation to optimization Newton
Remark. Optimization Newton applies the same linearization idea to \(F=\nabla f\). Numerical analysis emphasizes residuals, Jacobian solves, conditioning, and globalization for equations, while optimization adds objective decrease and second-order optimality.
Newton's iteration and quadratic convergence
For \(F:\mathbb R^n\to\mathbb R^n\), Newton's method solves \[ F'(x_k)s_k=-F(x_k),\qquad x_{k+1}=x_k+s_k. \]
Theorem (local quadratic convergence). Suppose \(F\in C^2\) near a root \(x^*\), \(F(x^*)=0\), and \(F'(x^*)\) is invertible. Then for \(x_0\) sufficiently close, \[ \|x_{k+1}-x^*\| \le C\|x_k-x^*\|^2. \]
Proof. Let \(e_k=x_k-x^*\). Taylor's formula gives \[ F(x_k)=F'(x_k)e_k+R_k,\qquad \|R_k\|\le L\|e_k\|^2/2 \] when \(F'\) is Lipschitz. Hence \[ e_{k+1} =e_k-F'(x_k)^{-1}F(x_k) =-F'(x_k)^{-1}R_k. \] Continuity of inversion bounds \(\|F'(x_k)^{-1}\|\) near \(x^*\), proving the claim.
Multiple roots
For a scalar root of multiplicity \(m\), \[ f(x)=(x-x^*)^mg(x),\qquad g(x^*)\ne0, \] ordinary Newton converges only linearly with factor \((m-1)/m\). If \(m\) is known, the modified step \[ x_{k+1}=x_k-m\frac{f(x_k)}{f'(x_k)} \] restores quadratic convergence.
Newton--Kantorovich certification
Theorem (schematic). Suppose \(\|F'(x_0)^{-1}F(x_0)\|\le\beta\), \[ \|F'(x_0)^{-1}(F'(x)-F'(y))\|\le L\|x-y\|, \] and \(h=\beta L\le1/2\) on a suitable ball. Then Newton iterates are well defined, remain in an explicit ball around \(x_0\), and converge to a unique root there.
The scalar majorant equation behind the proof bounds step lengths and invertibility through a Neumann series. This theorem converts local smoothness and a residual into a computable convergence region.
Globalization
Far from a root, a full Newton step can increase the residual or enter a region with singular Jacobian. Line search chooses \(\alpha_k\in(0,1]\) so \[ \|F(x_k+\alpha_ks_k)\|^2 \] decreases sufficiently. Trust-region methods minimize a local linear or quadratic model subject to \(\|s\|\le\Delta_k\), adapting \(\Delta_k\) by agreement between predicted and actual decrease.
Inexact Newton methods
Large systems solve the Newton equation only approximately: \[ \|F'(x_k)s_k+F(x_k)\| \le\eta_k\|F(x_k)\|. \] Uniform \(\eta_k<1\) gives local linear convergence; \(\eta_k\to0\) gives superlinear convergence, and \(\eta_k=O(\|F(x_k)\|)\) preserves quadratic convergence. Krylov tolerances should therefore tighten as the nonlinear iterate approaches the root.