Evaluation Metrics

Overall Fraction Can Hide a Slice

An overall held-out correct-row fraction can hide a weak slice. This lesson keeps slice rows and the combined row visible with exact fractions.

highlighted = computed this step

Slice A

Slice A has 4 rows: 4 correct and 0 wrong. Its correct-row fraction is 1.

44=1{4\over4}=1
Overall fraction can hide a sliceExact slice and overall correct-row fractions.slice summary tableslicerowscorrectwrongcorrect-row fractionSlice A4401Slice B4131/4overall8535/8overall row sums Slice A and Slice Boverall row is exact arithmetic; overall 5/8 does not show Slice B 1/4; does not prove future behavior; NOT training quality; NOTgeneralizationthreshold fixed first; metric alone is not a claim; NOT training

Slice B

Slice B also has 4 rows, but only 1 correct and 3 wrong. Its correct-row fraction is 1/4.

14=1/4{1\over4}=1/4
Overall fraction can hide a sliceExact slice and overall correct-row fractions.slice summary tableslicerowscorrectwrongcorrect-row fractionSlice A4401Slice B4131/4overall8535/8overall row sums Slice A and Slice Boverall row is exact arithmetic; overall 5/8 does not show Slice B 1/4; does not prove future behavior; NOT training quality; NOTgeneralizationthreshold fixed first; metric alone is not a claim; NOT training

Combine the rows

The overall row combines the slices: 8 rows, 5 correct, and 3 wrong, giving 5/8. The overall fraction 5/8 does not show that Slice B is 1/4. This is exact arithmetic on shown rows; it does not prove future behavior, training quality, or generalization.

58=5/8,Slice B=1/4{5\over8}=5/8,\quad \text{Slice B}=1/4
Overall fraction can hide a sliceExact slice and overall correct-row fractions.slice summary tableslicerowscorrectwrongcorrect-row fractionSlice A4401Slice B4131/4overall8535/8overall row sums Slice A and Slice Boverall row is exact arithmetic; overall 5/8 does not show Slice B 1/4; does not prove future behavior; NOT training quality; NOTgeneralizationthreshold fixed first; metric alone is not a claim; NOT training