Nearest Neighbors

kNN Regress by Mean

Predict a numeric value by averaging the target values of the k=3 nearest neighbors. Same squared-distance sort as knn-classify-majority; a second loop accumulates the target sum; pred = total/3. Library: sklearn.neighbors.KNeighborsRegressor(n_neighbors=3).fit(X, y) .predict([query]). RESULT: predicted value (rounded).

By hand

X_train=[[1,1],[2,1],[4,1],[8,7],[9,8],[7,9]], y=[10,20,35,80,90,85]. Query=[2,2]. Squared distances: [2,1]→1, [1,1]→2, [4,1]→5, others≥61. k=3 targets: 20, 10, 35. Mean = 65/3 ≈ 21.6667.

naive.py
Replay: real traced execution (multi-file project)
X_train = [[1,1], [2,1], [4,1], [8,7], [9,8], [7,9]]
y_train = [10, 20, 35, 80, 90, 85]
query = [2, 2]
n_train = len(X_train)
dists = []
for i in range(n_train):
    d0 = X_train[i][0] - query[0]
    d1 = X_train[i][1] - query[1]
    dists.append((d0*d0 + d1*d1, i))
dists.sort()
total = 0.0
for i in range(3):
    total = total + y_train[dists[i][1]]
pred = total / 3
print('RESULT:', round(pred, 4))

  1. X_train ← [[1, 1], [2, 1], [4, 1], [8, 7], [9, 8], [7, 9]]

    1X_train = [[1,1], [2,1], [4,1], [8,7], [9,8], [7,9]]2y_train = [10, 20, 35, 80, 90, 85]
    values this step[[1, 1], [2, 1], [4, 1], [8, 7], [9, 8], [7, 9]]X_train

  2. y_train ← [10, 20, 35, 80, 90, 85]

    1X_train = [[1,1], [2,1], [4,1], [8,7], [9,8], [7,9]]2y_train = [10, 20, 35, 80, 90, 85]3query = [2, 2]
    values this step[10, 20, 35, 80, 90, 85]y_train

  3. query ← [2, 2]

    2y_train = [10, 20, 35, 80, 90, 85]3query = [2, 2]4n_train = len(X_train)
    values this step[2, 2]query

  4. n_train ← 6

    3query = [2, 2]4n_train = len(X_train)5dists = []
    values this step6n_train

  5. dists ← []

    4n_train = len(X_train)5dists = []6for i in range(n_train):
    values this step[]dists

  6. i ← 0, d0 ← -1, d1 ← -1, dists ← [(2, 0)]

    pass 1 of 6
    5dists = []6for i in range(n_train):7    d0 = X_train[i][0] - query[0]8    d1 = X_train[i][1] - query[1]9    dists.append((d0*d0 + d1*d1, i))10dists.sort()
    values this step0i-1d0-1d1[] → [(2, 0)]dists
    All 6 passes — pass 1 is the card above
    passid0d1dists
    10-1-1[] → [(2, 0)]
    20 → 1-1 → 0—[(2, 0)] → [(2, 0), (1, 1)]
    31 → 20 → 2—[(2, 0), (1, 1)] → [(2, 0), (1, 1), (5, 2)]
    42 → 32 → 6-1 → 5[(2, 0), (1, 1), (5, 2)] → [(2, 0), (1, 1), (5, 2), (61, 3)]
    53 → 46 → 75 → 6[(2, 0), (1, 1), (5, 2), (61, 3)] → [(2, 0), (1, 1), (5, 2), (61, 3), (85, 4)]
    64 → 57 → 56 → 7[(2, 0), (1, 1), (5, 2), (61, 3), (85, 4)] → [(2, 0), (1, 1), (5, 2), (61, 3), (85, 4), (74, 5)]

  7. for i in range(n_train):

    5dists = []6for i in range(n_train):7    d0 = X_train[i][0] - query[0]

  8. dists ← [(1, 1), (2, 0), (5, 2), (61, 3), (74, 5), (85, 4)]

    9    dists.append((d0*d0 + d1*d1, i))10dists.sort()11total = 0.0
    values this step[(2, 0), (1, 1), (5, 2), (61, 3), (85, 4), (74, 5)] → [(1, 1), (2, 0), (5, 2), (61, 3), (74, 5), (85, 4)]dists

  9. total ← 0.0

    10dists.sort()11total = 0.012for i in range(3):
    values this step0.0total

  10. i ← 0, total ← 20.0

    pass 1 of 3
    11total = 0.012for i in range(3):13    total = total + y_train[dists[i][1]]14pred = total / 3
    values this step5 → 0i0.0 → 20.0total
    All 3 passes — pass 1 is the card above
    passitotal
    15 → 00.0 → 20.0
    20 → 120.0 → 30.0
    31 → 230.0 → 65.0

  11. for i in range(3):

    11total = 0.012for i in range(3):13    total = total + y_train[dists[i][1]]

  12. pred ← 21.666666666666668

    13    total = total + y_train[dists[i][1]]14pred = total / 315print('RESULT:', round(pred, 4))
    values this step21.666666666666668pred

  13. stdout ← RESULT: 21.6667

    14pred = total / 315print('RESULT:', round(pred, 4))
    values this stepRESULT: 21.6667stdout

With scikit-learn

KNeighborsRegressor(n_neighbors=3) averages the targets of the 3 nearest by Euclidean distance. kneighbors exposes the neighbors for verification.

library.py 
from sklearn.neighbors import KNeighborsRegressor
from dalib.display import set_display
set_display()

X_train = [[1,1], [2,1], [4,1], [8,7], [9,8], [7,9]]
y_train = [10, 20, 35, 80, 90, 85]
query = [2, 2]
reg = KNeighborsRegressor(n_neighbors=3)
reg.fit(X_train, y_train)
dists, indices = reg.kneighbors([query])
neighbors_y = [y_train[i] for i in indices[0]]
pred = float(reg.predict([query])[0])
print('k=3 targets:', neighbors_y)
print('distances:', [round(float(d), 4) for d in dists[0]])
print('RESULT:', round(pred, 4))

k=3 targets: [20, 10, 35]
distances: [1.0, 1.4142, 2.2361]
RESULT: 21.6667

Implementation notes

  • kNN regression averages the k nearest targets; kNN classification takes a majority vote. The distance ranking is identical — only the aggregation differs. Compare with knn-classify-majority (this chapter).
  • The 3 nearest boundary (sq=5 vs sq=61 for the 4th) is unambiguous, so naive and sklearn select the same neighbors.
  • pred = 21.6667: the three targets (10, 20, 35) span a range of 25; the mean is not dominated by any single neighbor.
  • Cross-reference: knn-classify-majority (this chapter) for the vote variant; euclidean-distance (this chapter) for the distance formula.