Folded-concave penalties
SCAD and MCP are designed to reduce the large-coefficient bias of the Lasso while keeping sparsity-inducing behavior near zero.
Definition. A separable penalized estimator minimizes \[ {1\over 2n}\|y-X\beta\|_2^2+\sum_{j=1}^p p_\lambda(|\beta_j|). \] SCAD and MCP choose \(p_\lambda\) concave on \([0,\infty)\), with derivative near zero equal to \(\lambda\) and derivative eventually zero for large coefficients.
Proposition. For an orthogonal design, a folded-concave penalty yields thresholding rules that set small coordinates to zero but apply less shrinkage to large coordinates than soft-thresholding.
Proof. Under orthogonality the objective separates coordinatewise. The derivative equation is \(\beta_j-z_j+p_\lambda'(|\beta_j|)\operatorname{sign}(\beta_j)=0\). Since \(p_\lambda'\) decreases to zero for large \(|\beta_j|\), large coordinates solve approximately \(\beta_j=z_j\), unlike Lasso where the shrinkage remains \(\lambda\).
Oracle property
Theorem. Under beta-min, local concavity, and suitable design conditions, folded-concave penalized estimators can have the oracle property: they recover the true support and estimate active coefficients as if the support were known.
Remark. The price is nonconvexity. Computation, initialization, and local minima become part of the statistical theorem rather than merely implementation details.
SCAD and MCP derivatives
Definition. SCAD and MCP are specified by decreasing derivatives. MCP has \(p'_\lambda(t)=(\lambda-t/a)_+\). SCAD has derivative \(\lambda\) for \(t\le\lambda\), \((a\lambda-t)/(a-1)\) for \(\lambda<t\le a\lambda\), and \(0\) after \(a\lambda\).
Proposition. The derivative becoming zero for large coefficients is the source of reduced asymptotic bias compared with Lasso.
Proof. The KKT equation on a nonzero coordinate is score plus \(p'_\lambda(|\beta_j|)\operatorname{sign}(\beta_j)=0\). If \(p'_\lambda\) vanishes for large signals, the large-coordinate equation matches the unpenalized score equation.
Integrated SCAD and MCP penalties
Definition. For \(a>2\), SCAD is \[ p_{\lambda,a}^{\mathrm{SCAD}}(t)= \begin{cases} \lambda t,&0\le t\le\lambda,\\ \dfrac{-t^2+2a\lambda t-\lambda^2}{2(a-1)}, &\lambda<t\le a\lambda,\\ \dfrac{(a+1)\lambda^2}{2},&t>a\lambda. \end{cases} \] For \(a>1\), MCP is \[ p_{\lambda,a}^{\mathrm{MCP}}(t)= \begin{cases} \lambda t-\dfrac{t^2}{2a},&0\le t\le a\lambda,\\ \dfrac{a\lambda^2}{2},&t>a\lambda. \end{cases} \]
Both penalties have slope \(\lambda\) at zero, creating thresholding, and zero slope beyond \(a\lambda\), removing shrinkage from sufficiently large signals.
Coordinatewise MCP thresholding
Proposition. For \[ \min_\beta\frac12(z-\beta)^2+ p_{\lambda,a}^{\mathrm{MCP}}(|\beta|), \] the global minimizer is \[ T_{\mathrm{MCP}}(z)= \begin{cases} 0,&|z|\le\lambda,\\ \dfrac{\operatorname{sign}(z)(|z|-\lambda)} {1-1/a},&\lambda<|z|\le a\lambda,\\ z,&|z|>a\lambda. \end{cases} \]
Proof. By symmetry take \(z\ge0\). On \(0<\beta<a\lambda\), stationarity gives \(\beta-z+\lambda-\beta/a=0\), hence \(\beta=(z-\lambda)/(1-1/a)\), valid exactly for \(\lambda<z<a\lambda\). Below \(\lambda\), the right derivative at zero is nonnegative and zero is optimal. Beyond \(a\lambda\), the penalty is constant and the minimizer is \(\beta=z\). Boundary comparison shows the pieces join continuously.
Difference-of-convex decomposition
Proposition. Every folded-concave penalty with \(p_\lambda'(0+)=\lambda\) can be written \[ p_\lambda(|t|)=\lambda|t|-q_\lambda(t), \] where \(q_\lambda\) is convex.
Proof. Define \(q_\lambda(t)=\lambda|t|-p_\lambda(|t|)\). For \(t>0\), \(q_\lambda'(t)=\lambda-p_\lambda'(t)\), which is nondecreasing because \(p_\lambda'\) is nonincreasing. Symmetry handles \(t<0\), and the one-sided derivatives satisfy convexity at zero.
Local linear approximation
Proposition. Concavity gives \[ p_\lambda(|\beta_j|) \le p_\lambda(|\beta_j^{(k)}|) +p_\lambda'(|\beta_j^{(k)}|) \{|\beta_j|-|\beta_j^{(k)}|\}. \] Minimizing this majorizer gives a weighted Lasso update with weight \(p_\lambda'(|\beta_j^{(k)}|)\), and the original objective decreases after every exact update.
Proof. A tangent line upper-bounds a concave function and touches it at the current iterate. The new surrogate value is no larger than the old one, and the true objective is bounded above by the surrogate at the new point.
Restricted curvature versus penalty concavity
Definition. The maximum concavity is \[ \mu_p=\sup_{t_1<t_2} -\frac{p_\lambda'(t_2)-p_\lambda'(t_1)}{t_2-t_1}. \] For MCP, \(\mu_p=1/a\); for SCAD, \(\mu_p=1/(a-1)\).
Theorem. If the empirical loss has restricted curvature \(\kappa>\mu_p\) on the relevant sparse cone, then the full objective retains restricted curvature \(\kappa-\mu_p\). Any stationary point in that region is locally unique and obeys oracle-style estimation bounds.
Proof. Along an admissible direction \(\Delta\), the loss contributes at least \(\kappa\|\Delta\|_2^2/2\) beyond its linearization. Penalty concavity can subtract at most \(\mu_p\|\Delta\|_2^2/2\). Their sum has quadratic growth \((\kappa-\mu_p)\|\Delta\|_2^2/2\).
The oracle estimator as a strict local minimum
Theorem. Let \(\widehat\beta^{\,O}\) minimize least squares on the true support \(S\) and be zero on \(S^c\). Suppose \[ \min_{j\in S}|\widehat\beta_j^{\,O}|>a\lambda,\qquad \left\|\frac{X_{S^c}^\top(Y-X\widehat\beta^{\,O})}{n}\right\|_\infty<\lambda, \] and the active Hessian is positive definite. Then \(\widehat\beta^{\,O}\) is a strict local minimum of SCAD- or MCP-penalized least squares.
Proof. On active coordinates the penalty derivative is zero because the coefficients exceed \(a\lambda\), so the oracle least-squares score vanishes. On inactive coordinates the score lies strictly inside the subgradient interval \([-\lambda,\lambda]\), preventing sufficiently small coordinate activation from decreasing the objective. Positive active curvature gives strict growth for perturbations within \(S\). Combining active and inactive directions proves strict local minimality.
Statistical and computational guarantees
A complete nonconvex theorem must identify which point an algorithm reaches. “There exists a local minimizer with the oracle property” does not imply that coordinate descent from zero finds it. Typical routes are:
- a convex initializer already lies in the attraction basin;
- restricted strong convexity dominates penalty concavity;
- a path-following algorithm starts at a large \(\lambda\) and decreases it gradually;
- every approximate stationary point in a controlled sparse region has the same statistical rate.
Remark. Reduced bias is not free. Folded concavity trades global convexity for weaker shrinkage. Larger \(a\) approaches the Lasso and improves convexity; smaller \(a\) removes bias sooner but makes the objective more nonconvex.