Two Stochastic Updates in Order - Stochastic Gradient Descent, Exactly

Stochastic updates are applied one after another. This lesson shows s2 first, then recomputes the s3 gradient from the newly updated parameters.

highlighted = computed this step

First update uses s2

Start at a=0, b=1. For sample s2, the prediction is 1, the residual is -1, and the gradient is (-1, -1). The update lands at (1/2, 3/2).

s2:\quad (a,b)=(0,1) \rightarrow (1/2,3/2)

Now the start is a=1/2, b=3/2. For sample s3, the prediction is recomputed as 7/2, the residual is 3/2, and the gradient is (3/2, 3).

s3:\quad \hat y=7/2,\quad g=(3/2,3)

Applying the second update gives (a,b)=(-1/4, 0). The second gradient is recomputed at the new parameters; it is not reused from the starting line.

(a,b)_{\text{after }s3}=(-1/4,0)