One Exact Step

Gradients Point Downhill

Gradient descent starts with exact gradients from the previous backprop book. This lesson reads those signs as directions for a one-step parameter move.

highlighted = computed this step

Gradients point downhill

A gradient records how the loss changes when a parameter moves. Descent uses that sign information by stepping against the gradient, parameter by parameter.

step against ∇w\text{step against } \nabla w
Gradient recapExact update table with required boundary note.gradient descent update, η=1/2parameteroldgradientη·gradientneww111-2-12w121-4-23b1-1-2-10w211001w22-100-1b20000v11-4-23v21001c0-2-11w_new = w_old - η·gradientone exact step with η=1/2; NOT convergence, NOT the right η, NOT learning

Backprop gave the exact gradients

From the exact backprop book, dL/dw11=-2, dL/dw12=-4, dL/db1=-2, and dL/dv1=-4.

dw11=−2,dw12=−4,db1=−2dw_{11}=-2,\quad dw_{12}=-4,\quad db_1=-2
Gradient recapExact update table with required boundary note.gradient descent update, η=1/2parameteroldgradientη·gradientneww111-2-12w121-4-23b1-1-2-10w211001w22-100-1b20000v11-4-23v21001c0-2-11w_new = w_old - η·gradientone exact step with η=1/2; NOT convergence, NOT the right η, NOT learning

Summary

The gradient register is exact. The next lesson adds the rational step size that turns those gradients into parameter moves.

∇wis exact here\nabla w\quad\text{is exact here}
Gradient recapExact update table with required boundary note.gradient descent update, η=1/2parameteroldgradientη·gradientneww111-2-12w121-4-23b1-1-2-10w211001w22-100-1b20000v11-4-23v21001c0-2-11w_new = w_old - η·gradientone exact step with η=1/2; NOT convergence, NOT the right η, NOT learning