Evaluation Metrics

Confusion Matrix Counts

Count TP, FP, FN, TN from binary true/predicted label lists. Loop over pairs: TP=true=1,pred=1; FP=true=0,pred=1; FN=true=1,pred=0; TN=true=0,pred=0. Library: sklearn confusion_matrix(labels=[0,1]) returns [[TN,FP],[FN,TP]]; extract TP=cm[1][1], FP=cm[0][1], FN=cm[1][0], TN=cm[0][0]. RESULT: (TP, FP, FN, TN) tuple.

Learning path

Prerequisite: Threshold Probabilities.

By hand

y_true=[1,0,1,1,0,1], y_pred=[1,1,0,1,0,0]. TP=2 (i=0,3); FP=1 (i=1); FN=2 (i=2,5); TN=1 (i=4).

naive.py
Replay: real traced execution (multi-file project)
y_true = [1, 0, 1, 1, 0, 1]
y_pred = [1, 1, 0, 1, 0, 0]
tp = 0
fp = 0
fn = 0
tn = 0
for i in range(len(y_true)):
    t = y_true[i]
    p = y_pred[i]
    if t == 1 and p == 1:
        tp = tp + 1
    elif t == 0 and p == 1:
        fp = fp + 1
    elif t == 1 and p == 0:
        fn = fn + 1
    else:
        tn = tn + 1
print('RESULT:', (tp, fp, fn, tn))

  1. y_true ← [1, 0, 1, 1, 0, 1]

    1y_true = [1, 0, 1, 1, 0, 1]2y_pred = [1, 1, 0, 1, 0, 0]
    values this step[1, 0, 1, 1, 0, 1]y_true

  2. y_pred ← [1, 1, 0, 1, 0, 0]

    1y_true = [1, 0, 1, 1, 0, 1]2y_pred = [1, 1, 0, 1, 0, 0]3tp = 0
    values this step[1, 1, 0, 1, 0, 0]y_pred

  3. tp ← 0

    2y_pred = [1, 1, 0, 1, 0, 0]3tp = 04fp = 0
    values this step0tp

  4. fp ← 0

    3tp = 04fp = 05fn = 0
    values this step0fp

  5. fn ← 0

    4fp = 05fn = 06tn = 0
    values this step0fn

  6. tn ← 0

    5fn = 06tn = 07for i in range(len(y_true)):
    values this step0tn

  7. i ← 0

    6tn = 07for i in range(len(y_true)):8    t = y_true[i]
    values this step0i

  8. t ← 1

    7for i in range(len(y_true)):8    t = y_true[i]9    p = y_pred[i]
    values this step1t

  9. p ← 1

    8t = y_true[i]9p = y_pred[i]10if t == 1 and p == 1:
    values this step1p

  10. if t == 1 and p == 1:

    9p = y_pred[i]10if t == 1 and p == 1:11    tp = tp + 1

  11. tp ← 1

    10if t == 1 and p == 1:11    tp = tp + 112elif t == 0 and p == 1:
    values this step0 → 1tp

  12. i ← 1

    6tn = 07for i in range(len(y_true)):8    t = y_true[i]
    values this step0 → 1i

  13. t ← 0

    7for i in range(len(y_true)):8    t = y_true[i]9    p = y_pred[i]
    values this step1 → 0t

  14. p = y_pred[i]

    8t = y_true[i]9p = y_pred[i]10if t == 1 and p == 1:

  15. if t == 1 and p == 1:

    9p = y_pred[i]10if t == 1 and p == 1:11    tp = tp + 1

  16. elif t == 0 and p == 1:

    11    tp = tp + 112elif t == 0 and p == 1:13    fp = fp + 1

  17. fp ← 1

    12elif t == 0 and p == 1:13    fp = fp + 114elif t == 1 and p == 0:
    values this step0 → 1fp

  18. i ← 2

    6tn = 07for i in range(len(y_true)):8    t = y_true[i]
    values this step1 → 2i

  19. t ← 1

    7for i in range(len(y_true)):8    t = y_true[i]9    p = y_pred[i]
    values this step0 → 1t

  20. p ← 0

    8t = y_true[i]9p = y_pred[i]10if t == 1 and p == 1:
    values this step1 → 0p

  21. if t == 1 and p == 1:

    9p = y_pred[i]10if t == 1 and p == 1:11    tp = tp + 1

  22. elif t == 0 and p == 1:

    11    tp = tp + 112elif t == 0 and p == 1:13    fp = fp + 1

  23. elif t == 1 and p == 0:

    13    fp = fp + 114elif t == 1 and p == 0:15    fn = fn + 1

  24. fn ← 1

    14elif t == 1 and p == 0:15    fn = fn + 116else:
    values this step0 → 1fn

  25. i ← 3

    6tn = 07for i in range(len(y_true)):8    t = y_true[i]
    values this step2 → 3i

  26. t = y_true[i]

    7for i in range(len(y_true)):8    t = y_true[i]9    p = y_pred[i]

  27. p ← 1

    8t = y_true[i]9p = y_pred[i]10if t == 1 and p == 1:
    values this step0 → 1p

  28. if t == 1 and p == 1:

    9p = y_pred[i]10if t == 1 and p == 1:11    tp = tp + 1

  29. tp ← 2

    10if t == 1 and p == 1:11    tp = tp + 112elif t == 0 and p == 1:
    values this step1 → 2tp

  30. i ← 4

    6tn = 07for i in range(len(y_true)):8    t = y_true[i]
    values this step3 → 4i

  31. t ← 0

    7for i in range(len(y_true)):8    t = y_true[i]9    p = y_pred[i]
    values this step1 → 0t

  32. p ← 0

    8t = y_true[i]9p = y_pred[i]10if t == 1 and p == 1:
    values this step1 → 0p

  33. if t == 1 and p == 1:

    9p = y_pred[i]10if t == 1 and p == 1:11    tp = tp + 1

  34. elif t == 0 and p == 1:

    11    tp = tp + 112elif t == 0 and p == 1:13    fp = fp + 1

  35. elif t == 1 and p == 0:

    13    fp = fp + 114elif t == 1 and p == 0:15    fn = fn + 1

  36. tn ← 1

    16    else:17        tn = tn + 118print('RESULT:', (tp, fp, fn, tn))
    values this step0 → 1tn

  37. i ← 5

    6tn = 07for i in range(len(y_true)):8    t = y_true[i]
    values this step4 → 5i

  38. t ← 1

    7for i in range(len(y_true)):8    t = y_true[i]9    p = y_pred[i]
    values this step0 → 1t

  39. p = y_pred[i]

    8t = y_true[i]9p = y_pred[i]10if t == 1 and p == 1:

  40. if t == 1 and p == 1:

    9p = y_pred[i]10if t == 1 and p == 1:11    tp = tp + 1

  41. elif t == 0 and p == 1:

    11    tp = tp + 112elif t == 0 and p == 1:13    fp = fp + 1

  42. elif t == 1 and p == 0:

    13    fp = fp + 114elif t == 1 and p == 0:15    fn = fn + 1

  43. fn ← 2

    14elif t == 1 and p == 0:15    fn = fn + 116else:
    values this step1 → 2fn

  44. for i in range(len(y_true)):

    6tn = 07for i in range(len(y_true)):8    t = y_true[i]

  45. stdout ← RESULT: (2, 1, 2, 1)

    17        tn = tn + 118print('RESULT:', (tp, fp, fn, tn))
    values this stepRESULT: (2, 1, 2, 1)stdout

With scikit-learn

confusion_matrix(y_true, y_pred, labels=[0,1]) returns a 2×2 matrix with layout [[TN,FP],[FN,TP]]: rows index true class (0 then 1), columns index predicted class (0 then 1).

library.py 
from sklearn.metrics import confusion_matrix
from dalib.display import set_display
set_display()

y_true = [1, 0, 1, 1, 0, 1]
y_pred = [1, 1, 0, 1, 0, 0]
cm = confusion_matrix(y_true, y_pred, labels=[0, 1])
print('cm:', cm.tolist())
tn, fp, fn, tp = int(cm[0][0]), int(cm[0][1]), int(cm[1][0]), int(cm[1][1])
print('RESULT:', (tp, fp, fn, tn))

cm: [[1, 1], [2, 2]]
RESULT: (2, 1, 2, 1)

Implementation notes

  • sklearn's layout [[TN,FP],[FN,TP]] indexes by (true_class, pred_class). With labels=[0,1]: cm[0][0]=TN, cm[0][1]=FP, cm[1][0]=FN, cm[1][1]=TP. Passing labels explicitly pins the row/column ordering regardless of which classes appear in the data.
  • TP and TN are correct predictions; FP is a false alarm (predicted positive, actually negative); FN is a miss (predicted negative, actually positive).
  • Accuracy = (TP+TN)/n; see Accuracy Score. Precision, Recall, and F1 covers metrics that handle class imbalance better than accuracy.
  • Threshold Probabilities shows how raising the decision threshold lowers FP at the cost of raising FN.