SGD, Exactly

Full-Batch Gradient

Full-batch gradient descent averages the exact per-sample gradients before applying the update.

highlighted = computed this step

Per-sample gradients

For one row, the gradient of one-half squared error is residual times 1 for a and residual times x for b.

gi=(ri, rixi)g_i=(r_i,\ r_ix_i)
Full-batch gradientAverage exact gradients across all samples.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)one exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning

Full batch averages them

The full-batch gradient averages all 3 sample gradients: a-gradient -1, -1, 0 average to -2/3.

gfull,a=(−1 + −1 + 0)/3=−2/3g_{\text{full},a}=(-1\,+\,-1\,+\,0)/3=-2/3
Full-batch gradientAverage exact gradients across all samples.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)one exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning

The full gradient

The exact full-batch gradient is (-2/3, -1/3). No decimal approximation is used.

gfull=(−2/3,−1/3)g_{\text{full}}=(-2/3,-1/3)
Full-batch gradientAverage exact gradients across all samples.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)one exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning