Ridge, Exactly

The Setup

Ridge starts from the least-squares fit. This book uses a through-origin one-weight model so every change is visible as one exact denominator change.

highlighted = computed this step

The setup

Use a through-origin line y≈w*x on the exact points (1, 2) and (2, 3). With one weight, least squares uses the exact ratio sum xy over sum x squared.

y≈wx,(x,y)=(1,2), (2,3)y\approx w x,\quad (x,y)=(1, 2),\ (2, 3)
Least squares through the originExact unregularized through-origin fit.ridge solve, λ=0shown pointx*yx^2p1=(1,2)21p2=(2,3)64Σxy=8; Σx^2=5Σx^2+λ=5; w(λ)=8/5through-origin ridge fit; λ is chosen before the solve; shrinkage is mechanical, NOT afuture-data claim

Exact sums

The shown data give sum xy=8 and sum x squared=5. So the unregularized weight is 8/5.

∑xy=8,∑x2=5,w=8/5\sum xy=8,\quad \sum x^2=5,\quad w=8/5
Least squares through the originExact unregularized through-origin fit.ridge solve, λ=0shown pointx*yx^2p1=(1,2)21p2=(2,3)64Σxy=8; Σx^2=5Σx^2+λ=5; w(λ)=8/5through-origin ridge fit; λ is chosen before the solve; shrinkage is mechanical, NOT afuture-data claim

Why ridge enters

A large or sensitive weight can be discouraged by adding a penalty. This is a mechanical change to the arithmetic, not a claim about future data.

change the objective, then recompute exactly\text{change the objective, then recompute exactly}
Least squares through the originExact unregularized through-origin fit.ridge solve, λ=0shown pointx*yx^2p1=(1,2)21p2=(2,3)64Σxy=8; Σx^2=5Σx^2+λ=5; w(λ)=8/5through-origin ridge fit; λ is chosen before the solve; shrinkage is mechanical, NOT afuture-data claim