Optimization begins with two logically separate questions: does an optimizer exist, and how can it be recognized or computed? Even in finite dimensions an infimum need not be attained, a stationary point need not minimize anything, and a local minimizer need not be globally useful. The distinctions below remain important when the objective is a likelihood, risk, energy, or variational posterior criterion.
Objectives, feasible sets, and attainment
Definition. Let \(f:\mathbb R^d\to(-\infty,+\infty]\) be an extended-real-valued function. Its effective domain is \[ \operatorname{dom}f=\{x:f(x)<+\infty\}. \] Minimizing \(f\) over a set \(C\subset\mathbb R^d\) is equivalent to minimizing \(f+\iota_C\) over \(\mathbb R^d\), where \(\iota_C(x)=0\) on \(C\) and \(+\infty\) off \(C\). The optimal value and solution set are \[ f^\star=\inf_{x\in C}f(x),\qquad X^\star=\operatorname*{arg\,min}_{x\in C}f(x). \]
The notation deliberately permits \(X^\star=\varnothing\). For example, \(\inf_{x\in\mathbb R}e^x=0\), but no finite \(x\) attains the value. Existence therefore cannot be inferred merely from boundedness below.
Definition. A feasible point \(x^\star\in C\) is a local minimizer if there is \(r>0\) such that \(f(x^\star)\le f(x)\) for every \(x\in C\cap B(x^\star,r)\). It is a strict local minimizer if the inequality is strict for every \(x\ne x^\star\) in that neighborhood. It is isolated if it is the only local minimizer in some neighborhood.
Strictness and isolation are different. The function \(f(x)=x^4\) has a strict isolated minimum at zero although \(f''(0)=0\); the function \(f(x,y)=x^2\) has global minima along the whole line \(x=0\), none of which is isolated.
Feasible directions and the tangent cone
Definition. The Bouligand tangent cone of \(C\) at \(x\in C\) is \[ T_C(x)=\left\{v:\ \exists\,t_n\downarrow0,\ x_n\in C,\ x_n\to x,\quad \frac{x_n-x}{t_n}\to v\right\}. \] When \(C\) is convex, \(T_C(x)=\overline{\{t(y-x):t\ge0,\ y\in C\}}\).
First-order necessary condition. Suppose \(f\) is differentiable at a local minimizer \(x^\star\) relative to \(C\). Then \[ \langle\nabla f(x^\star),v\rangle\ge0 \qquad\text{for every }v\in T_C(x^\star). \] In particular, if \(x^\star\) is an interior point of \(C\), then \(\nabla f(x^\star)=0\).
Proof. For \(v\in T_C(x^\star)\), choose \(x_n\in C\) and \(t_n\downarrow0\) as in the definition. Local minimality gives \(f(x_n)-f(x^\star)\ge0\) for all large \(n\). Differentiability yields \[ \frac{f(x_n)-f(x^\star)}{t_n} = \left\langle\nabla f(x^\star), \frac{x_n-x^\star}{t_n}\right\rangle +\frac{o(\|x_n-x^\star\|)}{t_n}. \] The last term tends to zero because \(\|x_n-x^\star\|/t_n\to\|v\|\). Passing to the limit proves the inequality. At an interior point, \(T_C(x^\star)=\mathbb R^d\); applying the inequality to \(v\) and \(-v\) forces the gradient to vanish.
Second-order tests and their limits
Second-order conditions. Let \(f\) be twice differentiable in a neighborhood of an unconstrained stationary point \(x^\star\).
(i) If \(x^\star\) is a local minimizer, then \(v^\top\nabla^2f(x^\star)v\ge0\) for every \(v\).
(ii) If \(\nabla^2f(x^\star)\) is positive definite, then \(x^\star\) is a strict isolated local minimizer.
Proof. For fixed \(v\), Taylor's formula gives \[ f(x^\star+tv)-f(x^\star) =\frac{t^2}{2}v^\top\nabla^2f(x^\star)v+o(t^2). \] Local nonnegativity of the left side for both signs of small \(t\) proves (i). For (ii), let \(\lambda_{\min}(\nabla^2f(x^\star))=2c>0\). Continuity of the Hessian gives \(\nabla^2f(x)\succeq cI\) near \(x^\star\). Taylor's theorem along the segment from \(x^\star\) to \(x\) then gives \(f(x)\ge f(x^\star)+\frac c2\|x-x^\star\|^2\), with strict inequality away from \(x^\star\).
Example. At the origin, \(f(x,y)=x^2-y^2\) has zero gradient and indefinite Hessian, so it is a saddle. The function \(g(x)=x^3\) has zero gradient and zero Hessian but is not a minimum. The function \(h(x)=x^4\) has the same first two derivatives at zero and is a strict minimum. Thus positive definiteness is sufficient, not necessary, and a semidefinite Hessian is inconclusive.
Why convexity changes the problem
Local minima of convex functions are global. If \(C\) and \(f:C\to\mathbb R\) are convex, every local minimizer of \(f\) on \(C\) is a global minimizer.
Proof. If \(x^\star\) were locally but not globally minimizing, choose \(y\in C\) with \(f(y)<f(x^\star)\). For \(t\in(0,1)\), convexity gives \[ f((1-t)x^\star+ty) \le(1-t)f(x^\star)+tf(y)<f(x^\star). \] Taking \(t\downarrow0\) produces feasible points arbitrarily close to \(x^\star\) with smaller objective value, contradicting local minimality.
This is the first reason convexity is central: local information becomes global. The next pages develop the geometry of convex feasible sets and the first-order calculus of convex objectives.