Stochastic Gradient Descent, Exactly

SGD, Exactly

• Tiny Linear Data
• Full-Batch Gradient
• One Sample Gradient
• Two Stochastic Updates in Order
• One Epoch Is One Pass
• Same Samples, Different Order
• Mini-Batch Gradient
• Side-by-Side Updates
• Why the Steps Differ
• What SGD Is and Is Not