Exact Sparsity

Exact Sparsity

At the threshold value, the soft-thresholded numerator becomes exactly zero. This is the toy example's clean sparsity moment.

highlighted = computed this step

The sparsity threshold

At λ=28, the numerator max(28 minus 28, 0) is exactly 0.

max⁡(28 − 28,0)=0\max(28\,-\,28,0)=0
Exact sparsityLambda twenty-eight hits zero.lasso solve, λ=28shown pointx*yx^2p1=(1,2)21p2=(2,4)84p3=(3,6)189Σxy=28; Σx^2=14max(Σxy-λ,0)=0; w=0through-origin lasso fit; λ is chosen before the solve; L1 can hit exact zero; NOTconvergence; NOT general feature selection; NOT generalization

The weight is exactly zero

Therefore w(28)=0/14=0. This is exact sparsity: the coefficient is equal to zero.

w(28)=0w(28)=0
Exact sparsityLambda twenty-eight hits zero.lasso solve, λ=28shown pointx*yx^2p1=(1,2)21p2=(2,4)84p3=(3,6)189Σxy=28; Σx^2=14max(Σxy-λ,0)=0; w=0through-origin lasso fit; λ is chosen before the solve; L1 can hit exact zero; NOTconvergence; NOT general feature selection; NOT generalization

Summary

Lasso does more than shrink toward zero in this toy setup. Once λ reaches the source correlation, the exact solution hits zero.

λ=∑xy⇒w=0\lambda=\sum xy\quad\Rightarrow\quad w=0
Exact sparsityLambda twenty-eight hits zero.lasso solve, λ=28shown pointx*yx^2p1=(1,2)21p2=(2,4)84p3=(3,6)189Σxy=28; Σx^2=14max(Σxy-λ,0)=0; w=0through-origin lasso fit; λ is chosen before the solve; L1 can hit exact zero; NOTconvergence; NOT general feature selection; NOT generalization