SGD, Exactly

What SGD Is and Is Not

The finale keeps the boundary visible: one stochastic update is not a convergence or generalization claim.

highlighted = computed this step

What is exact

This book computes one full-batch update and one sample update from the same tiny rational data, same start, and same eta=1/2.

same data, same start, same η\text{same data, same start, same }\eta
Full batch versus one sampleTwo exact updates with the same eta.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)same eta=1/2, different gradient sourceupdategrad agrad bnew anew bfull batch-2/3-1/31/37/6sample s2-1-11/23/2new = old - eta*gradientone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning

What is outside

The rendered lesson does not choose an epoch order, does not prove convergence, and does not claim generalization.

one step≠convergence proof\text{one step}\ne\text{convergence proof}
Full batch versus one sampleTwo exact updates with the same eta.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)same eta=1/2, different gradient sourceupdategrad agrad bnew anew bfull batch-2/3-1/31/37/6sample s2-1-11/23/2new = old - eta*gradientone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning

What SGD is and is not

SGD here means: use one chosen sample gradient for one exact update. It is NOT convergence, NOT generalization, and NOT learning by itself.

one exact stochastic step; deferred claims explicit\text{one exact stochastic step; deferred claims explicit}
Full batch versus one sampleTwo exact updates with the same eta.per-sample gradients; full batch is the averagesampleresidualgrad agrad bs1-1-10s2-1-1-1s3000full gradient=(-2/3, -1/3)same eta=1/2, different gradient sourceupdategrad agrad bnew anew bfull batch-2/3-1/31/37/6sample s2-1-11/23/2new = old - eta*gradientone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning