One Weight Update - Backpropagation, Exactly

One exact gradient becomes one exact parameter update when the assigned step size is applied. This lesson isolates w11 so the signed change and new weight can be checked before returning to the full annotated graph.

highlighted = computed this step

Old weight, gradient, step size

Focus on w11. The old weight is 1, the gradient dL/dw11 is -2, and the assigned step size is eta=1/4. These three exact values are the full source for this one update.

w_{11}=1,\quad \frac{dL}{dw_{11}}=-2,\quad \eta=1/4

Compute the signed change

The update rule subtracts eta times the gradient. Here the signed change is - 1/4*(-2)=1/2. Because the gradient is negative, subtracting eta times that gradient moves the weight up by an exact half.

-\;1/4\cdot(-2)=1/2

Add the change

Add the signed change to the old weight: 1 + 1/2 = 3/2. So this one displayed update sends w11 from 1 to 3/2.

w_{11}^{new}=1\;+\;1/2\;=\;3/2

Boundary

This is one toy parameter update only. It is not training, not convergence, not learning, and no generalization claim is made.

\text{one parameter update; not a training claim}