SGD, Exactly

Side-by-Side Updates

The same start and same eta produce two different exact rows because the gradient source differs.

highlighted = computed this step

Same start, same eta

Both rows start at a=0, b=1 and use eta=1/2. Only the gradient source changes.

η=1/2\eta=1/2
Full batch versus one sampleTwo exact updates with the same eta.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)same eta=1/2, different gradient sourceupdategrad agrad bnew anew bfull batch-2/3-1/31/37/6sample s2-1-11/23/2new = old - eta*gradientone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning

Full-batch result

The full-batch row uses gradient (-2/3, -1/3) and lands at (1/3, 7/6).

(a,b)full=(1/3,7/6)(a,b)_{\text{full}}=(1/3,7/6)
Full batch versus one sampleTwo exact updates with the same eta.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)same eta=1/2, different gradient sourceupdategrad agrad bnew anew bfull batch-2/3-1/31/37/6sample s2-1-11/23/2new = old - eta*gradientone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning

Sample result

The stochastic row uses sample s2 and lands at (1/2, 3/2).

(a,b)s2=(1/2,3/2)(a,b)_{s2}=(1/2,3/2)
Full batch versus one sampleTwo exact updates with the same eta.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)same eta=1/2, different gradient sourceupdategrad agrad bnew anew bfull batch-2/3-1/31/37/6sample s2-1-11/23/2new = old - eta*gradientone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning