Accuracy Can Hide

Label Counts Are Visible

Start by counting the visible N and P labels.

highlighted = computed this step

Show the label mix

The displayed table has 5 rows. Rows A through D have label N, and row E has label P.

rows=A,B,C,D,E\text{rows}=A,B,C,D,E
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth

Count the N rows

There are 4 N rows.

N=4N=4
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth

Count the P rows

There is 1 P row.

P=1P=1
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth