The inverse and implicit function theorems say that a nonlinear map with invertible linearization behaves locally like that linearization.
Local linear invertibility
The key hypothesis is invertibility of the derivative at the base point.
Definition. A \(C^1\) map \(F:U\subset\mathbb R^d\to\mathbb R^d\) has nonsingular derivative at \(x_0\) if the matrix \(DF(x_0)\) is invertible.
Lemma. If \(DF(x_0)\) is invertible and \(DF\) is continuous, then \(DF(x)\) remains invertible for all \(x\) in a sufficiently small neighborhood of \(x_0\).
Proof. Put \(A=DF(x_0)\). Then \[ DF(x)=A\left[I+A^{-1}(DF(x)-A)\right]. \] Continuity lets us choose a neighborhood on which \[ \|A^{-1}(DF(x)-A)\|\le q<1. \] For every operator \(E\) with \(\|E\|<1\), \[ (I+E)^{-1}=\sum_{k=0}^{\infty}(-E)^k \] in operator norm. Hence \(DF(x)\) is invertible there, and \[ \|DF(x)^{-1}\|\le\frac{\|A^{-1}\|}{1-q}. \]
Inverse function theorem
A \(C^1\) map with invertible derivative is locally a \(C^1\) coordinate change.
Theorem. If \(F\in C^1\) near \(x_0\) and \(DF(x_0)\) is invertible, then there are neighborhoods \(U_0\) of \(x_0\) and \(V_0\) of \(F(x_0)\) such that \(F:U_0\to V_0\) is bijective and \(F^{-1}\in C^1\).
Proof. Translate so that \(x_0=F(x_0)=0\), and replace \(F\) by \(DF(0)^{-1}F\). It remains to consider \[ F(x)=x+R(x),\qquad R(0)=0,\qquad DR(0)=0. \] Choose \(r>0\) so that \(\|DR(x)\|\le q<1/2\) on \(\overline B(0,r)\). The mean value inequality gives \[ \|R(x)-R(z)\|\le q\|x-z\|. \] For \(\|y\|\le(1-q)r\), define \(T_y(x)=y-R(x)\). Since \(\|R(x)\|\le q\|x\|\), \(T_y\) maps \(\overline B(0,r)\) into itself, and it is a \(q\)-contraction. Banach's fixed-point theorem gives a unique \(x\) in the ball with \(F(x)=y\).
Also, \[ \|F(x)-F(z)\| \ge(1-q)\|x-z\|, \] so \(F\) is injective on the ball and its local inverse is Lipschitz. Let \(y=F(x)\), \(k=F(x+h)-F(x)\). Since \(\|h\|\le(1-q)^{-1}\|k\|\), \[ k=DF(x)h+o(\|h\|) \quad\Longrightarrow\quad h=DF(x)^{-1}k+o(\|k\|). \] Thus \(D(F^{-1})(y)=DF(x)^{-1}\). Continuity of \(DF\) and continuity of matrix inversion make this derivative continuous.
Corollary (local bi-Lipschitz estimate). After shrinking \(U_0\), constants \(0<c\le C<\infty\) exist such that \[ c\|x-z\|\le\|F(x)-F(z)\|\le C\|x-z\|, \qquad x,z\in U_0. \] Thus the theorem supplies not only a local inverse but quantitative stability of that inverse.
Implicit function theorem
Equations can be solved locally for variables whose derivative block is invertible.
Theorem. Let \(G:\mathbb R^p\times\mathbb R^q\to\mathbb R^q\) be \(C^1\), \(G(a,b)=0\), and \(D_yG(a,b)\) invertible. Then locally the equation \(G(x,y)=0\) is equivalent to \(y=\varphi(x)\) for a \(C^1\) function \(\varphi\).
Proof. Define \[ F(x,y)=(x,G(x,y)). \] At \((a,b)\), \[ DF(a,b)= \begin{pmatrix} I_p&0\\ D_xG(a,b)&D_yG(a,b) \end{pmatrix}, \] which is invertible because it is block triangular with invertible diagonal blocks. The inverse function theorem gives a local \(C^1\) inverse. Since the first component of \(F\) is \(x\), that inverse must have the form \[ F^{-1}(u,v)=(u,\psi(u,v)). \] Setting \(\varphi(u)=\psi(u,0)\) gives \(G(u,\varphi(u))=0\). Conversely, local injectivity of \(F\) shows that every nearby solution of \(G(x,y)=0\) has \(y=\varphi(x)\).
Derivative of the implicit map
Differentiating the constraint gives the sensitivity formula.
Corollary. Under the implicit function theorem hypotheses, \[ D\varphi(x)=-[D_yG(x,\varphi(x))]^{-1}D_xG(x,\varphi(x)). \]
Proof. Differentiate the identity \(G(x,\varphi(x))=0\). The chain rule gives \(D_xG+D_yG\,D\varphi=0\). Since \(D_yG\) is invertible, solve for \(D\varphi\).
Estimating equations and argmin sensitivity
Many estimators are local solutions of score equations or first-order optimality conditions.
Remark. Suppose \(\Psi(\theta,\eta)=0\) defines \(\theta=\theta(\eta)\). If \(D_\theta\Psi\) is nonsingular, then \[ D\theta(\eta) =-[D_\theta\Psi(\theta(\eta),\eta)]^{-1} D_\eta\Psi(\theta(\eta),\eta). \] For \(\Psi=\nabla_\theta M\), the inverse derivative is the inverse Hessian. This formula is the deterministic ancestor of influence-function calculations and local expansions for \(M\)- and \(Z\)-estimators; see Statistical inference.
Example (a regular level set). Let \(G(x,y)=x^2+y^2-1\). At a point \((a,b)\) on the unit circle with \(b\ne0\), \(D_yG(a,b)=2b\ne0\), so the circle is locally the graph \(y=\varphi(x)\), and \[ \varphi'(x)=-\frac{2x}{2\varphi(x)}=-\frac{x}{\varphi(x)}. \] The failure at \((\pm1,0)\) does not mean the circle is singular; it means that there one should solve for \(x\) as a function of \(y\).