Evaluation Metrics

Accuracy Can Hide Misses

A high overall fraction can sit beside missed positives when the shown dataset is imbalanced. This lesson keeps the exact grid, overall-rate denominator, and positive recall denominator visible.

highlighted = computed this step

Start with the shown grid

This shown set has 10 rows: 2 actual positives and 8 actual negatives. The outputs miss both positives, so TP=0 and FN=2.

TP=0,FN=2,FP=0,TN=8TP=0,\quad FN=2,\quad FP=0,\quad TN=8
Accuracy can hide missesExact imbalanced grid.missed positives gridoutputpositivenegativeactualpositivenegativeTP0FN2FP0TN8accuracy=4/5positive recall=0one displayed grid only; does not provefuture performance or training behaviorthreshold fixed first; metric alone is not a claim; NOT training

Accuracy can still look high

The overall correct-row rate counts every correct row. Here the 8 true negatives make the rate 4/5 even though no positive row was caught.

accuracy⁡=0+810=4/5\operatorname{accuracy}={0\mathbin{+}8\over10}=4/5
Accuracy can hide missesExact imbalanced grid.missed positives gridoutputpositivenegativeactualpositivenegativeTP0FN2FP0TN8accuracy=4/5positive recall=0one displayed grid only; does not provefuture performance or training behaviorthreshold fixed first; metric alone is not a claim; NOT training

Recall exposes the misses

Positive recall reads only the actual positive row group. The exact recall is 0 because TP=0 out of 2 actual positives. This is one displayed grid only; it does not prove future performance or training behavior.

recall⁡+=02=0\operatorname{recall}_{+}={0\over2}=0
Accuracy can hide missesExact imbalanced grid.missed positives gridoutputpositivenegativeactualpositivenegativeTP0FN2FP0TN8accuracy=4/5positive recall=0one displayed grid only; does not provefuture performance or training behaviorthreshold fixed first; metric alone is not a claim; NOT training