Surrogate losses replace discontinuous decision losses by convex or smooth objectives. Calibration is the condition that optimizing the surrogate also optimizes the target decision problem.
Surrogate risk
A surrogate is useful only if its risk remembers the original task.
Definition. For binary \(Y\in\{-1,1\}\), a margin surrogate has risk \[ R_\phi(f)=\mathbb E\,\phi(Yf(X)). \] It is classification-calibrated if every sequence with \(R_\phi(f_n)\to\inf_f R_\phi(f)\) also has zero-one risk approaching the Bayes risk.
Remark. Convexity helps computation; calibration is the statistical condition that prevents the convex problem from solving the wrong task.
Conditional calibration
Calibration can be checked pointwise through the conditional risk.
Theorem. Let \(\eta=P(Y=1\mid X=x)\). For a convex differentiable margin loss \(\phi\), classification calibration holds if \(\phi'(0)<0\).
Proof. The conditional surrogate risk is \[ C_\eta(t)=\eta\phi(t)+(1-\eta)\phi(-t). \] At \(t=0\), \(C_\eta'(0)=(2\eta-1)\phi'(0)\). If \(\eta>1/2\), then \(C_\eta'(0)<0\), so a minimizer must prefer positive scores; if \(\eta<1/2\), the sign reverses. Thus the surrogate minimizer has the Bayes sign.
Excess-risk transfer
A calibrated surrogate gives a quantitative link between surrogate excess risk and classification excess risk.
Theorem. For every classification-calibrated convex margin loss, there exists a nondecreasing function \(\psi\) with \(\psi(0)=0\) such that \[ \psi(R(g_f)-R^*)\le R_\phi(f)-R_\phi^*. \]
Proof. Define the best conditional surrogate risk under the correct sign and under the wrong sign. The gap between these two conditional optima is positive away from \(\eta=1/2\) by calibration. Taking the convex lower envelope of the resulting conditional gap yields \(\psi\). Integrating the conditional inequality gives the displayed bound.
Logistic and hinge losses
The most common surrogates have different geometry but the same calibrated direction.
Example. The logistic loss \(\log(1+e^{-u})\) is smooth and strictly convex in the margin. The hinge loss \((1-u)_+\) is nonsmooth and produces sparse support-vector solutions in linear RKHS formulations.
Remark. The logistic surrogate estimates conditional probabilities naturally through log-odds, while hinge loss focuses on the decision boundary and margin.
Conditional Bayes risks of standard surrogates
Proposition. For \(C_\eta(t)=\eta\phi(t)+(1-\eta)\phi(-t)\): \[ \begin{array}{c|c|c} \phi(u)&t_\eta^*&C_\eta^*\\ \hline \log(1+e^{-u})& \log\{\eta/(1-\eta)\}& -\eta\log\eta-(1-\eta)\log(1-\eta)\\ e^{-u}& \frac12\log\{\eta/(1-\eta)\}& 2\sqrt{\eta(1-\eta)}\\ (1-u)_+& \operatorname{sign}(2\eta-1)& 2\min(\eta,1-\eta). \end{array} \]
Proof. Differentiate the smooth conditional risks and solve the first-order equations. For hinge loss, examine the three linear regions \(t\le-1\), \(-1\le t\le1\), and \(t\ge1\); the minimum occurs at \(1\) when \(\eta>1/2\), at \(-1\) when \(\eta<1/2\), and throughout \([-1,1]\) at equality.
Proper scoring rules and probability estimation
Definition. A binary loss \(L(q,y)\), \(q\in[0,1]\), is proper if \[ \eta L(q,1)+(1-\eta)L(q,0) \] is minimized at \(q=\eta\), and strictly proper if this minimizer is unique.
Theorem. Every differentiable strictly proper binary loss has conditional regret equal to a Bregman divergence: \[ C_\eta(q)-C_\eta(\eta)=D_{-H}(\eta,q), \] where \(H(\eta)=\inf_qC_\eta(q)\) is the concave Bayes risk.
Proof outline. Properness implies that the tangent line to the concave function \(H\) at \(q\) encodes the two losses: \(L(q,1)=H(q)+(1-q)H'(q)\) and \(L(q,0)=H(q)-qH'(q)\). Substitution gives \[ C_\eta(q)-H(\eta) =H(q)+H'(q)(\eta-q)-H(\eta) =D_{-H}(\eta,q). \] Strict concavity of \(H\) makes the divergence positive away from \(q=\eta\).
Example. Log loss has \[ C_\eta(q)-C_\eta(\eta) =\operatorname{KL}\{\operatorname{Bern}(\eta)\| \operatorname{Bern}(q)\}. \] Pinsker's inequality then yields \[ 2(\eta-q)^2\le C_\eta(q)-C_\eta(\eta). \] Integrating gives \(2\|\widehat\eta-\eta\|_{L^2(P_X)}^2\) as a lower bound for logistic probability regret.
An explicit logistic regret transfer
Theorem. Let \(q_f(x)=\{1+e^{-f(x)}\}^{-1}\), and classify by \(g_f=\mathbf1\{f\ge0\}\). Then \[ R_{01}(g_f)-R_{01}^* \le\sqrt{2\{R_{\log}(f)-R_{\log}^*\}}. \]
Proof. The plug-in bound gives \(R_{01}(g_f)-R_{01}^*\le2\mathbb E|q_f-\eta|\). Cauchy-Schwarz and the pointwise Pinsker bound give \[ 2\mathbb E|q_f-\eta| \le2\{\mathbb E(q_f-\eta)^2\}^{1/2} \le\sqrt{2\{R_{\log}(f)-R_{\log}^*\}}. \]
Hinge regret equals classification regret
Proposition. For every score \(f\), \[ R_{01}(g_f)-R_{01}^* \le R_{\mathrm{hinge}}(f)-R_{\mathrm{hinge}}^*. \]
Proof. Fix \(\eta\). If \(f\) has the wrong sign, its conditional hinge risk is minimized under that sign at \(t=0\), where it equals \(1\). The unrestricted conditional optimum is \(2\min(\eta,1-\eta)=1-|2\eta-1|\). Thus wrong-sign hinge regret is at least \(|2\eta-1|\), exactly the conditional zero-one regret. Correct-sign points contribute nonnegative hinge regret and zero classification regret. Integrate.
Composite losses and links
Definition. A proper composite loss writes \[ \ell(y,t)=L\{y,\psi^{-1}(t)\}, \] where \(L\) is proper and \(\psi:(0,1)\to\mathbb R\) is an invertible link. Logistic loss uses the logit link; squared probability loss uses the identity link.
The link changes optimization geometry but not the probability functional elicited by the underlying proper loss. A badly conditioned link can make gradients vanish or explode near the boundary even though statistical calibration remains valid.
Multiclass calibration
Proposition. For \(Y\in\{1,\ldots,K\}\), softmax cross-entropy \[ \ell(s,y)=-s_y+\log\sum_{k=1}^Ke^{s_k} \] has conditional minimizers satisfying \[ \frac{e^{s_k}}{\sum_je^{s_j}}=\eta_k. \] Therefore \(\operatorname*{argmax}_ks_k\) is a Bayes classifier.
Proof. The conditional risk is \(-\sum_k\eta_ks_k+\log\sum_ke^{s_k}\). Its gradient is \(-\eta+\operatorname{softmax}(s)\), so any stationary point has the displayed probabilities. The Hessian is the covariance matrix of a categorical distribution, hence positive semidefinite; translation \(s\mapsto s+c\mathbf1\) is the only nonidentifiability.
A calibrated loss can still generalize poorly
Remark. Calibration is a population statement: it says that vanishing surrogate regret forces vanishing task regret. It does not guarantee that empirical surrogate risk can be estimated uniformly over a rich class, nor that an optimizer finds a near-global minimum. A complete argument needs three pieces: \[ \text{optimization error} +\text{surrogate estimation error} +\text{calibration transfer}. \] Confusing these levels is a common source of unjustified claims.