One Exact Step

Same Gradient, Different Eta

One parameter row keeps the same old value and gradient while three rational eta choices change the exact move size.

highlighted = computed this step

Pin one row

Use the same parameter row every time: w11 has old value 1 and gradient -2. Only eta changes across the rows.

w11=1,∇w11=−2w_{11}=1,\quad \nabla w_{11}=-2
Same Gradient, Different EtaExact eta comparison table.same gradient, different etaparameteroldgradientetaeta*gradientneww111-21/4-1/23/2w111-21/2-12w111-21-23same old value and gradient; eta changes the move sizesame old value and gradient; chosen eta values only; NOT convergence, NOT the righteta, NOT the right η, NOT training, NOT learningsame old value and gradient; chosen eta values only; NOT convergence, NOT the righteta, NOT the right η, NOT training, NOT learning

Eta changes the move

The three chosen eta values are 1/4, 1/2, and 1. Multiplying each by the same gradient gives -1/2, -1, and -2.

1/4⋅(−2)=−1/2,1/2⋅(−2)=−11/4\cdot(-2)=-1/2,\quad 1/2\cdot(-2)=-1
Same Gradient, Different EtaExact eta comparison table.same gradient, different etaparameteroldgradientetaeta*gradientneww111-21/4-1/23/2w111-21/2-12w111-21-23same old value and gradient; eta changes the move sizesame old value and gradient; chosen eta values only; NOT convergence, NOT the righteta, NOT the right η, NOT training, NOT learningsame old value and gradient; chosen eta values only; NOT convergence, NOT the righteta, NOT the right η, NOT training, NOT learning

Subtract the move

The update is old minus eta times gradient. The new values are 3/2, 2, and 3.

1−(−1/2)=3/2,1−(−2)=31-(-1/2)=3/2,\quad 1-(-2)=3
Same Gradient, Different EtaExact eta comparison table.same gradient, different etaparameteroldgradientetaeta*gradientneww111-21/4-1/23/2w111-21/2-12w111-21-23same old value and gradient; eta changes the move sizesame old value and gradient; chosen eta values only; NOT convergence, NOT the righteta, NOT the right η, NOT training, NOT learningsame old value and gradient; chosen eta values only; NOT convergence, NOT the righteta, NOT the right η, NOT training, NOT learning

Boundary

This compares one displayed parameter and three chosen step sizes. It is NOT convergence, NOT the right eta, NOT training, and NOT learning.

same gradient, three exact move sizes\text{same gradient, three exact move sizes}
Same Gradient, Different EtaExact eta comparison table.same gradient, different etaparameteroldgradientetaeta*gradientneww111-21/4-1/23/2w111-21/2-12w111-21-23same old value and gradient; eta changes the move sizesame old value and gradient; chosen eta values only; NOT convergence, NOT the righteta, NOT the right η, NOT training, NOT learningsame old value and gradient; chosen eta values only; NOT convergence, NOT the righteta, NOT the right η, NOT training, NOT learning