Exact Sparsity

Ridge vs Lasso

Ridge and lasso use the same data here, but the penalties behave differently. The comparison shows ridge nonzero and lasso exactly zero.

highlighted = computed this step

Same data, same λ

Compare ridge and lasso on the same data at λ=28. Lasso uses the thresholded numerator; ridge puts λ in the denominator.

λ=28\lambda=28
Ridge versus lassoExact contrast at lambda twenty-eight.ridge versus lasso at λ=28penaltyformulaweightlasso L1max(28-λ,0)/140ridge L228/(14+λ)2/3lasso hits zero; ridge stays nonzerothrough-origin lasso fit; λ is chosen before the solve; L1 can hit exact zero; NOTconvergence; NOT general feature selection; NOT generalization

The exact contrast

At this λ, lasso gives 0. Ridge gives 2/3. Ridge shrinks but stays nonzero; lasso reaches exact zero.

wlasso=0,wridge=2/3w_{\text{lasso}}=0,\quad w_{\text{ridge}}=2/3
Ridge versus lassoExact contrast at lambda twenty-eight.ridge versus lasso at λ=28penaltyformulaweightlasso L1max(28-λ,0)/140ridge L228/(14+λ)2/3lasso hits zero; ridge stays nonzerothrough-origin lasso fit; λ is chosen before the solve; L1 can hit exact zero; NOTconvergence; NOT general feature selection; NOT generalization

Summary

This is the headline difference from the ridge book. Lasso's absolute-value corner can produce an exact zero coefficient.

ridge nonzero; lasso zero\text{ridge nonzero; lasso zero}
Ridge versus lassoExact contrast at lambda twenty-eight.ridge versus lasso at λ=28penaltyformulaweightlasso L1max(28-λ,0)/140ridge L228/(14+λ)2/3lasso hits zero; ridge stays nonzerothrough-origin lasso fit; λ is chosen before the solve; L1 can hit exact zero; NOTconvergence; NOT general feature selection; NOT generalization