Accuracy Can Hide

Accuracy Is Four Out of Five

Compute the exact correct-row fraction from correct rows.

highlighted = computed this step

Compute the correct-row fraction

There are 4 correct rows out of 5 total rows.

accuracy=45\text{accuracy}={4\over5}
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth

The fraction is high

The correct-row fraction is 4 over 5. The table still shows row E was missed.

4/54/5
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth

The fraction is not the whole story

The single accuracy fraction does not show which class was missed.

keep the row table visible\text{keep the row table visible}
Class Imbalance ExactlyExact row counts for one always-N baseline.label countslabelcountN4P1always-N baseline on one imbalanced tablerowtruepredcorrect?cellANNyesTNBNNyesTNCNNyesTNDNNyesTNEPNmissFNconfusion counts with P as positivecountvalueTP0FN1FP0TN4metric fractionsmetricfractionaccuracy4/5positive recall0/1 = 0accuracy can be high while the lone P is missedone pinned imbalanced table; one always-N baselinethe lone P is missed; this is exact row countingdoes not claim: NOT fairness solved; NOT future performanceNOT calibrated; NOT safe; NOT best metric; NOT accuracy guaranteeNOT generalization; NOT probability truth