Accuracy Can Hide

Always-N Baseline

Apply one simple baseline to every displayed row.

highlighted = computed this step

Use the always-N baseline

Every displayed prediction is N. That means rows A, B, C, and D match their true label.

y^A=y^B=y^C=y^D=N\hat y_A=\hat y_B=\hat y_C=\hat y_D=N
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth

The P row is missed

Row E has true label P, but the always-N baseline puts N in the output cell.

E: P→NE:\ P \rightarrow N
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth

Mark correct and missed rows

Rows A through D are correct. Row E is the lone miss.

correct rows=A,B,C,D\text{correct rows}=A,B,C,D
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth