Newton's method solves the first-order equation \(\nabla f(x)=0\) by linearizing the gradient. Equivalently, it exactly minimizes the quadratic Taylor model. Near a nondegenerate solution this removes the leading linear error and leaves a quadratic remainder.
The local quadratic model
Definition. At \(x\), let \(g=\nabla f(x)\) and \(H=\nabla^2f(x)\). The quadratic model is \[ m_x(p)=f(x)+g^\top p+\frac12p^\top Hp. \] If \(H\) is nonsingular, the Newton direction solves \(Hp_N=-g\), and the full Newton update is \(x^+=x+p_N\).
If \(H\succ0\), \(p_N\) uniquely minimizes the model and \(g^\top p_N=-g^\top H^{-1}g<0\). If \(H\) is indefinite, the stationary point of the model may be a saddle and need not be a descent direction.
The quadratic error equation
Local quadratic convergence. Let \(f\in C^2\) near \(x^\star\), with \(\nabla f(x^\star)=0\). Suppose \(H_\star=\nabla^2f(x^\star)\) is invertible and the Hessian is locally \(M\)-Lipschitz. Then there is a neighborhood of \(x^\star\) in which the Newton update is well-defined and \[ \|x_{k+1}-x^\star\| \le C\|x_k-x^\star\|^2. \] Hence every sufficiently close initial point converges quadratically.
Proof. Write \(e=x-x^\star\) and \(H(x)=\nabla^2f(x)\). The fundamental theorem of calculus gives \[ \nabla f(x) =\int_0^1H(x^\star+te)e\,\mathrm{d}t. \] Therefore \[ e^+ =e-H(x)^{-1}\nabla f(x) =H(x)^{-1}\int_0^1 [H(x)-H(x^\star+te)]e\,\mathrm{d}t. \] Continuity and invertibility of \(H_\star\) give \(\|H(x)^{-1}\|\le K\) near \(x^\star\). Hessian Lipschitzness then yields \[ \|e^+\| \le KM\int_0^1(1-t)\,\mathrm{d}t\,\|e\|^2 =\frac{KM}{2}\|e\|^2. \] Shrink the neighborhood so that the update remains inside it; induction proves convergence and the quadratic bound.
Quadratic objectives terminate in one step. For \(f(x)=\frac12x^\top Qx-b^\top x\), \(Q\) nonsingular, \[ x^+=x-Q^{-1}(Qx-b)=Q^{-1}b=x^\star. \] Newton is exact because the Taylor model is the objective itself.
Affine invariance
Affine covariance of the step. Let \(x=Az+b\) with \(A\) invertible and \(\widetilde f(z)=f(Az+b)\). If \(p_z\) is the Newton direction for \(\widetilde f\), then \(Ap_z\) is the Newton direction for \(f\).
Proof. The chain rule gives \[ \nabla\widetilde f(z)=A^\top\nabla f(x),\qquad \nabla^2\widetilde f(z)=A^\top\nabla^2f(x)A. \] Thus \[ p_z=-(A^\top HA)^{-1}A^\top g =-A^{-1}H^{-1}g, \] and multiplying by \(A\) gives \(Ap_z=-H^{-1}g=p_x\).
Gradient descent lacks this property because its identity metric does not transform with the coordinates. Newton's Hessian supplies the transforming metric automatically.
Newton decrement and local accuracy
When \(H(x)\succ0\), define the Newton decrement \[ \lambda(x)^2=\nabla f(x)^\top H(x)^{-1}\nabla f(x) =-\,\nabla f(x)^\top p_N. \] It measures the predicted model decrease: \(m_x(0)-m_x(p_N)=\lambda(x)^2/2\). For self-concordant convex functions, \(\lambda(x)\) also controls the true objective gap and determines when full Newton steps are safe.
Why globalization is necessary
Counterexample. For \(f(x)=x^4/4-x^2/2\), the Newton update is \[ x^+=x-\frac{x^3-x}{3x^2-1}. \] Near \(x=\pm1/\sqrt3\), the denominator is almost zero and the full step can be arbitrarily large. At points where the Hessian is negative, the Newton direction can increase \(f\). Local quadratic convergence says nothing about such starting points.
Backtracking line search dampens the direction until sufficient descent is observed. Trust-region methods minimize the quadratic model only inside a region where it is believed. Modified Newton methods also regularize or factorize the Hessian to enforce a descent direction.