Linear Regression

Gradient Step (One Weight)

Perform ONE gradient-descent step for a 1D model y_pred=w·x (no bias). Gradient of MSE w.r.t. w: grad = (2/n)Σ((w·xᵢ−yᵢ)·xᵢ). Update: w_new = w − lr·grad. Library: NumPy vectorized gradient confirms the loop. RESULT: (gradient, w_new) rounded.

By hand

With NumPy

np.sum((w*x - y) * x) computes Σ((w·xᵢ−yᵢ)·xᵢ) vectorized.

naive.py 
x = [1, 2, 4]
y = [2, 4, 8]
w = 1.0
lr = 0.1
n = len(x)
grad = 0.0
for i in range(n):
    err = w * x[i] - y[i]
    grad = grad + err * x[i]
grad = (2 / n) * grad
w_new = w - lr * grad
print('RESULT:', (round(grad, 4), round(w_new, 4)))
library.py 
import numpy as np
from dalib.display import set_display
set_display()

x = np.array([1, 2, 4])
y = np.array([2, 4, 8])
w = 1.0
lr = 0.1
n = len(x)
grad = round(float((2 / n) * np.sum((w * x - y) * x)), 4)
w_new = round(w - lr * grad, 4)
print('gradient:', grad)
print('RESULT:', (grad, w_new))

gradient: -14.0
RESULT: (-14.0, 2.4)

Honesty

This lesson shows a single gradient step computed exactly, on a tiny pinned sample. The updated weight is arithmetically correct and reproducible, but one step on a handful of points illustrates the update rule, not a trained model — the resulting weight is not a fitted estimate and warrants no inference. Real fitting needs many steps (or a closed-form solve), an adequate sample, and a held-out check before any prediction is trustworthy.

Implementation notes

  • ONE step only — full gradient descent iterates this update until convergence. Closed-form normal-equation-1d (this chapter) reaches the exact optimum in one computation; gradient descent approximates it iteratively.
  • The gradient is negative (MSE decreases as w increases from 1.0 toward the optimum); subtracting a negative gradient raises w.
  • The true optimum for this data is w*=2.0 (Σxᵢyᵢ/Σxᵢ²=42/21=2.0). One step with lr=0.1 overshoots to 2.4 — illustrating how learning rate choice affects convergence.
  • No bias term: y_pred=w·x. Adding an intercept b requires a second update: b_new = b − lr·(2/n)Σ(w·xᵢ+b−yᵢ).
  • The gradient formula (2/n)Σ((w·xᵢ−yᵢ)·xᵢ) is the derivative of MSE = (1/n)Σ(w·xᵢ−yᵢ)² with respect to w by the chain rule.