Differentiability is linear approximation at small scales. Taylor's theorem refines this approximation by recording higher-order local information.
Derivative as best linear approximation
In one dimension, the derivative is the coefficient of the first-order local model.
Definition. Let \(x\) be an interior point of an interval \(I\). A function \(f:I\to\mathbb R\) is differentiable at \(x\) if there is a number \(A\) such that \[ f(x+h)=f(x)+Ah+r(h),\qquad \frac{r(h)}{|h|}\longrightarrow0. \] The unique coefficient \(A\) is denoted \(f'(x)\). Equivalently, \[ f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}h. \]
Proposition. If \(f\) is differentiable at \(x\), then \(f(x+h)=f(x)+f'(x)h+o(h)\) as \(h\to0\), and \(f\) is continuous at \(x\).
Proof. The definition says \((f(x+h)-f(x))/h=f'(x)+r(h)\) with \(r(h)\to0\). Multiplying by \(h\) gives the expansion. Since \(f(x+h)-f(x)=h(f'(x)+r(h))\to0\), continuity follows.
Algebra and chain rules
Differentiability is stable under the usual algebraic operations.
Theorem. If \(f,g\) are differentiable at \(x\), then \(f+g\), \(fg\), and \(f/g\) where \(g(x)\ne0\), are differentiable at \(x\). If \(g\) is differentiable at \(x\) and \(f\) is differentiable at \(g(x)\), then \((f\circ g)'(x)=f'(g(x))g'(x)\).
Proof. Addition and scalar multiplication follow by adding the first-order expansions. For the product, \[ \begin{aligned} f(x+h)g(x+h)-f(x)g(x) &=f(x)\bigl(g(x+h)-g(x)\bigr)\\ &\quad+g(x)\bigl(f(x+h)-f(x)\bigr)\\ &\quad+\bigl(f(x+h)-f(x)\bigr)\bigl(g(x+h)-g(x)\bigr). \end{aligned} \] The last term is \(o(h)\) because each increment is \(O(h)\), giving \((fg)'=f'g+fg'\). Since \(g(x+h)\to g(x)\ne0\), \[ \frac{g(x+h)^{-1}-g(x)^{-1}}h =-\frac{1}{g(x+h)g(x)}\frac{g(x+h)-g(x)}h \longrightarrow-\frac{g'(x)}{g(x)^2}. \] This and the product rule give the quotient rule.
For the chain rule set \(k(h)=g(x+h)-g(x)\). Write \[ f(g(x)+u)-f(g(x))=f'(g(x))u+u\eta(u), \] where \(\eta(u)\to0\), defining \(\eta(0)=0\). Since \(k(h)=g'(x)h+o(h)=O(h)\), \[ f(g(x+h))-f(g(x)) =f'(g(x))k(h)+k(h)\eta(k(h)) =f'(g(x))g'(x)h+o(h). \]
Taylor theorem with remainder
Higher derivatives give polynomial approximations and quantitative errors.
Theorem. If \(f\in C^{n+1}\) on an interval containing \(a\) and \(x\), then \[ f(x)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^k+\frac{f^{(n+1)}(\xi)}{(n+1)!}(x-a)^{n+1} \] for some \(\xi\) between \(a\) and \(x\).
Proof. Assume \(x\ne a\) and let \[ T_n(t)=\sum_{k=0}^n\frac{f^{(k)}(a)}{k!}(t-a)^k,\qquad C=\frac{f(x)-T_n(x)}{(x-a)^{n+1}}. \] The function \(G(t)=f(t)-T_n(t)-C(t-a)^{n+1}\) satisfies \(G(x)=0\) and \[ G(a)=G'(a)=\cdots=G^{(n)}(a)=0. \] Rolle's theorem applied successively to \(G,G',\ldots,G^{(n)}\) yields \(\xi\) between \(a\) and \(x\) with \(G^{(n+1)}(\xi)=0\). Since \[ G^{(n+1)}(\xi)=f^{(n+1)}(\xi)-(n+1)!C, \] the asserted formula follows.
Corollary (quantitative remainder). If \(|f^{(n+1)}(t)|\le M\) between \(a\) and \(x\), then \[ \left|f(x)-\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^k\right| \le\frac{M}{(n+1)!}|x-a|^{n+1}. \] For example, taking \(f(x)=e^x\) at \(a=0\) proves convergence of its Taylor polynomials uniformly on every bounded interval.
Integral remainder
Theorem. If \(f^{(n)}\) is absolutely continuous between \(a\) and \(x\), then \[ f(x)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^k +\frac1{n!}\int_a^x f^{(n+1)}(t)(x-t)^n\,\mathrm{d}t. \]
Proof. For \(n=0\), this is the fundamental theorem of calculus. If it holds for \(n-1\), integrate the last term by parts: \[ \frac1{(n-1)!}\int_a^x f^{(n)}(t)(x-t)^{n-1}\,\mathrm{d}t =\frac{f^{(n)}(a)}{n!}(x-a)^n +\frac1{n!}\int_a^x f^{(n+1)}(t)(x-t)^n\,\mathrm{d}t. \] Induction proves the formula. Unlike the Lagrange form, this remainder also works naturally with weak integrability assumptions.
Peano form
The Peano remainder is often cleaner for asymptotics.
Theorem. If \(f\) has \(n\) derivatives near \(a\) and \(f^{(n)}\) is continuous at \(a\), then \[ f(a+h)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}h^k+o(h^n). \]
Proof. Subtract the Taylor polynomial and write the remainder as \(R_n(h)\). Repeated Cauchy mean-value arguments give, for \(h\ne0\), a point \(\xi_h\) between \(a\) and \(a+h\) such that \[ \frac{R_n(h)}{h^n} =\frac{f^{(n)}(\xi_h)-f^{(n)}(a)}{n!}. \] Since \(\xi_h\to a\) and \(f^{(n)}\) is continuous at \(a\), the right side tends to zero.
Quadratic expansions in statistical asymptotics
The first derivative gives the local linear response; the second derivative controls local curvature and identifiability.
Remark. If \(M_n'(\widehat\theta_n)=0\), expansion around \(\theta_0\) gives \[ 0=M_n'(\theta_0)+M_n''(\theta_0)(\widehat\theta_n-\theta_0) +o\bigl(|\widehat\theta_n-\theta_0|\bigr). \] After proving that \(M_n''(\theta_0)\) stays away from zero and controlling the remainder uniformly, this identity turns a limit theorem for the score \(M_n'(\theta_0)\) into one for the estimator. The multivariate and stochastic versions are developed in Statistical inference.