One Exact Step

Choosing the Step Size

The step size is chosen before the update. This book uses one rational value and computes the resulting move exactly.

highlighted = computed this step

Choosing the step size

Eta is a choice. Here the chosen value is the exact rational 1/2, so every update remains in the exact register.

η=1/2\eta=1/2
Choosing the step sizeExact update table with required boundary note.gradient descent update, η=1/2parameteroldgradientη·gradientneww111-2-12w121-4-23b1-1-2-10w211001w22-100-1b20000v11-4-23v21001c0-2-11w_new = w_old - η·gradientone exact step with η=1/2; NOT convergence, NOT the right η, NOT learning

The scale changes the move

A larger eta would multiply the same gradients into larger moves. A smaller eta would multiply them into smaller moves. This surface fixes one value and computes that case exactly.

η∇wsets the move size\eta\nabla w\quad\text{sets the move size}
Choosing the step sizeExact update table with required boundary note.gradient descent update, η=1/2parameteroldgradientη·gradientneww111-2-12w121-4-23b1-1-2-10w211001w22-100-1b20000v11-4-23v21001c0-2-11w_new = w_old - η·gradientone exact step with η=1/2; NOT convergence, NOT the right η, NOT learning

Boundary

This is one exact step. The render makes no claim about the right eta or how many steps should be used.

one chosen η, one exact update\text{one chosen }\eta,\text{ one exact update}
Choosing the step sizeExact update table with required boundary note.gradient descent update, η=1/2parameteroldgradientη·gradientneww111-2-12w121-4-23b1-1-2-10w211001w22-100-1b20000v11-4-23v21001c0-2-11w_new = w_old - η·gradientone exact step with η=1/2; NOT convergence, NOT the right η, NOT learning