SGD, Exactly

Tiny Linear Data

The SGD comparison starts with one tiny rational linear-regression dataset and a fixed starting line.

highlighted = computed this step

Tiny linear model

The model is y_hat = a + b x. The start is a=0 and b=1, before either update is applied.

y^=a+bx,a=0,b=1\hat y=a+bx,\quad a=0,\quad b=1
Tiny dataExact residuals before either update.tiny data, start a=0, b=1samplexypredictionresiduals1010-1s2121-1s32220residual = prediction - yone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning

The three data rows

The displayed points are (0, 1), (1, 2), and (2, 2). Every prediction and residual in the table is recomputed from those rows.

(x,y)∈{(0,1),(1,2),(2,2)}(x,y)\in\{(0, 1),(1, 2),(2, 2)\}
Tiny dataExact residuals before either update.tiny data, start a=0, b=1samplexypredictionresiduals1010-1s2121-1s32220residual = prediction - yone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning

Residuals at the start

At the start, the residuals are -1, -1, and 0. The residual convention is prediction minus y.

r=(y^−y)=(−1,−1,0)r=(\hat y-y)=(-1,-1,0)
Tiny dataExact residuals before either update.tiny data, start a=0, b=1samplexypredictionresiduals1010-1s2121-1s32220residual = prediction - yone exact step only; sample is chosen before the update; NOT convergence; NOT generalization;NOT learning