Ridge, Exactly

The Ridge Penalty

Regularization enters as a penalty term added to the squared-error objective. In the one-weight setup, that changes the normal equation without leaving exact arithmetic.

highlighted = computed this step

The ridge penalty

Ridge adds λ*w squared to the squared-error objective. In the through-origin one-weight model, that adds λ to the denominator.

squared error+λw2\text{squared error}+\lambda w^2
Ridge penaltyExact ridge denominator for lambda one.ridge solve, λ=1shown pointx*yx^2p1=(1,2)21p2=(2,3)64Σxy=8; Σx^2=5Σx^2+λ=6; w(λ)=4/3through-origin ridge fit; λ is chosen before the solve; shrinkage is mechanical, NOT afuture-data claim

Modified normal equation

The exact normal equation becomes w times (sum x squared plus λ) equals sum xy. Solving gives w(λ)=sum xy divided by (sum x squared plus λ).

w(∑x2+λ)=∑xy,w(λ)=∑xy∑x2+λw(\sum x^2+\lambda)=\sum xy,\quad w(\lambda)={\sum xy\over \sum x^2+\lambda}
Ridge penaltyExact ridge denominator for lambda one.ridge solve, λ=1shown pointx*yx^2p1=(1,2)21p2=(2,3)64Σxy=8; Σx^2=5Σx^2+λ=6; w(λ)=4/3through-origin ridge fit; λ is chosen before the solve; shrinkage is mechanical, NOT afuture-data claim

Summary

Regularization has not introduced a non-exact step. The displayed λ is a chosen exact value, and the weight is still a Fraction.

λ∈Q⇒w(λ)∈Q\lambda\in\mathbb{Q}\quad\Rightarrow\quad w(\lambda)\in\mathbb{Q}
Ridge penaltyExact ridge denominator for lambda one.ridge solve, λ=1shown pointx*yx^2p1=(1,2)21p2=(2,3)64Σxy=8; Σx^2=5Σx^2+λ=6; w(λ)=4/3through-origin ridge fit; λ is chosen before the solve; shrinkage is mechanical, NOT afuture-data claim