Linear Regression

Gradient Step (One Weight)

Perform ONE gradient-descent step for a 1D model y_pred=w·x (no bias). Gradient of MSE w.r.t. w: grad = (2/n)Σ((w·xᵢ−yᵢ)·xᵢ). Update: w_new = w − lr·grad. Library: NumPy vectorized gradient confirms the loop. RESULT: (gradient, w_new) rounded.

By hand

x=[1,2,4], y=[2,4,8]. w=1.0, lr=0.1. Errors: −1,−2,−4. Σ(err·x)=−1−4−16=−21. grad=(2/3)(−21)=−14.0. w_new=1.0−0.1·(−14.0)=2.4.

naive.py
Replay: real traced execution (multi-file project)
x = [1, 2, 4]
y = [2, 4, 8]
w = 1.0
lr = 0.1
n = len(x)
grad = 0.0
for i in range(n):
    err = w * x[i] - y[i]
    grad = grad + err * x[i]
grad = (2 / n) * grad
w_new = w - lr * grad
print('RESULT:', (round(grad, 4), round(w_new, 4)))

  1. x ← [1, 2, 4]

    1x = [1, 2, 4]2y = [2, 4, 8]
    values this step[1, 2, 4]x

  2. y ← [2, 4, 8]

    1x = [1, 2, 4]2y = [2, 4, 8]3w = 1.0
    values this step[2, 4, 8]y

  3. w ← 1.0

    2y = [2, 4, 8]3w = 1.04lr = 0.1
    values this step1.0w

  4. lr ← 0.1

    3w = 1.04lr = 0.15n = len(x)
    values this step0.1lr

  5. n ← 3

    4lr = 0.15n = len(x)6grad = 0.0
    values this step3n

  6. grad ← 0.0

    5n = len(x)6grad = 0.07for i in range(n):
    values this step0.0grad

  7. i ← 0

    6grad = 0.07for i in range(n):8    err = w * x[i] - y[i]
    values this step0i

  8. err ← -1.0

    7for i in range(n):8    err = w * x[i] - y[i]9    grad = grad + err * x[i]
    values this step-1.0err

  9. grad ← -1.0

    8    err = w * x[i] - y[i]9    grad = grad + err * x[i]10grad = (2 / n) * grad
    values this step0.0 → -1.0grad

  10. i ← 1

    6grad = 0.07for i in range(n):8    err = w * x[i] - y[i]
    values this step0 → 1i

  11. err ← -2.0

    7for i in range(n):8    err = w * x[i] - y[i]9    grad = grad + err * x[i]
    values this step-1.0 → -2.0err

  12. grad ← -5.0

    8    err = w * x[i] - y[i]9    grad = grad + err * x[i]10grad = (2 / n) * grad
    values this step-1.0 → -5.0grad

  13. i ← 2

    6grad = 0.07for i in range(n):8    err = w * x[i] - y[i]
    values this step1 → 2i

  14. err ← -4.0

    7for i in range(n):8    err = w * x[i] - y[i]9    grad = grad + err * x[i]
    values this step-2.0 → -4.0err

  15. grad ← -21.0

    8    err = w * x[i] - y[i]9    grad = grad + err * x[i]10grad = (2 / n) * grad
    values this step-5.0 → -21.0grad

  16. for i in range(n):

    6grad = 0.07for i in range(n):8    err = w * x[i] - y[i]

  17. grad ← -14.0

    9    grad = grad + err * x[i]10grad = (2 / n) * grad11w_new = w - lr * grad
    values this step-21.0 → -14.0grad

  18. w_new ← 2.4000000000000004

    10grad = (2 / n) * grad11w_new = w - lr * grad12print('RESULT:', (round(grad, 4), round(w_new, 4)))
    values this step2.4000000000000004w_new

  19. stdout ← RESULT: (-14.0, 2.4)

    11w_new = w - lr * grad12print('RESULT:', (round(grad, 4), round(w_new, 4)))
    values this stepRESULT: (-14.0, 2.4)stdout

With NumPy

np.sum((w*x - y) * x) computes Σ((w·xᵢ−yᵢ)·xᵢ) vectorized.

library.py 
import numpy as np
from dalib.display import set_display
set_display()

x = np.array([1, 2, 4])
y = np.array([2, 4, 8])
w = 1.0
lr = 0.1
n = len(x)
grad = round(float((2 / n) * np.sum((w * x - y) * x)), 4)
w_new = round(w - lr * grad, 4)
print('gradient:', grad)
print('RESULT:', (grad, w_new))

gradient: -14.0
RESULT: (-14.0, 2.4)

Honesty

This lesson shows a single gradient step computed exactly, on a tiny pinned sample. The updated weight is arithmetically correct and reproducible, but one step on a handful of points illustrates the update rule, not a trained model — the resulting weight is not a fitted estimate and warrants no inference. Real fitting needs many steps (or a closed-form solve), an adequate sample, and a held-out check before any prediction is trustworthy.

Implementation notes

  • ONE step only — full gradient descent iterates this update until convergence. Closed-form normal-equation-1d (this chapter) reaches the exact optimum in one computation; gradient descent approximates it iteratively.
  • The gradient is negative (MSE decreases as w increases from 1.0 toward the optimum); subtracting a negative gradient raises w.
  • The true optimum for this data is w*=2.0 (Σxᵢyᵢ/Σxᵢ²=42/21=2.0). One step with lr=0.1 overshoots to 2.4 — illustrating how learning rate choice affects convergence.
  • No bias term: y_pred=w·x. Adding an intercept b requires a second update: b_new = b − lr·(2/n)Σ(w·xᵢ+b−yᵢ).
  • The gradient formula (2/n)Σ((w·xᵢ−yᵢ)·xᵢ) is the derivative of MSE = (1/n)Σ(w·xᵢ−yᵢ)² with respect to w by the chain rule.