Nearest Neighbors

kNN Regress by Mean

Predict a numeric value by averaging the target values of the k=3 nearest neighbors. Same squared-distance sort as knn-classify-majority; a second loop accumulates the target sum; pred = total/3. Library: sklearn.neighbors.KNeighborsRegressor(n_neighbors=3).fit(X, y) .predict([query]). RESULT: predicted value (rounded).

By hand

With scikit-learn

KNeighborsRegressor(n_neighbors=3) averages the targets of the 3 nearest by Euclidean distance. kneighbors exposes the neighbors for verification.

naive.py 
X_train = [[1,1], [2,1], [4,1], [8,7], [9,8], [7,9]]
y_train = [10, 20, 35, 80, 90, 85]
query = [2, 2]
n_train = len(X_train)
dists = []
for i in range(n_train):
    d0 = X_train[i][0] - query[0]
    d1 = X_train[i][1] - query[1]
    dists.append((d0*d0 + d1*d1, i))
dists.sort()
total = 0.0
for i in range(3):
    total = total + y_train[dists[i][1]]
pred = total / 3
print('RESULT:', round(pred, 4))
library.py 
from sklearn.neighbors import KNeighborsRegressor
from dalib.display import set_display
set_display()

X_train = [[1,1], [2,1], [4,1], [8,7], [9,8], [7,9]]
y_train = [10, 20, 35, 80, 90, 85]
query = [2, 2]
reg = KNeighborsRegressor(n_neighbors=3)
reg.fit(X_train, y_train)
dists, indices = reg.kneighbors([query])
neighbors_y = [y_train[i] for i in indices[0]]
pred = float(reg.predict([query])[0])
print('k=3 targets:', neighbors_y)
print('distances:', [round(float(d), 4) for d in dists[0]])
print('RESULT:', round(pred, 4))

k=3 targets: [20, 10, 35]
distances: [1.0, 1.4142, 2.2361]
RESULT: 21.6667

Implementation notes

  • kNN regression averages the k nearest targets; kNN classification takes a majority vote. The distance ranking is identical — only the aggregation differs. Compare with knn-classify-majority (this chapter).
  • The 3 nearest boundary (sq=5 vs sq=61 for the 4th) is unambiguous, so naive and sklearn select the same neighbors.
  • pred = 21.6667: the three targets (10, 20, 35) span a range of 25; the mean is not dominated by any single neighbor.
  • Cross-reference: knn-classify-majority (this chapter) for the vote variant; euclidean-distance (this chapter) for the distance formula.