Evaluation Metrics

One Row Moves a Small Metric

A metric on a small held-out slice can move a lot when one row changes. The same one-row change moves a larger slice less.

highlighted = computed this step

Tiny held-out slice

In a tiny held-out slice with 5 rows, the correct-row fraction moves quickly. Before the row changes, 4 are correct and 1 is wrong, so the fraction is 4/5. After one correct row changes to wrong, the fraction is 3/5.

45=4/5→35=3/5{4\over5}=4/5\quad\to\quad {3\over5}=3/5
One row moves a small metricExact accuracy movement by held-out slice size.one row metric moveslicerowsbefore correct/wrongbefore accafter correct/wrongafter accmovetiny held-out slice54/14/53/23/51/5larger held-out slice5040/104/539/1139/501/50one correct row changes to wrong in each slicearithmetic sensitivity to sample size only; one pinned held-out slice; does not prove future performance; NOT training behavior; NOTuncertainty intervals; NOT generalizationthreshold fixed first; metric alone is not a claim; NOT training

Larger held-out slice

In a larger held-out slice with 50 rows, the same one-row change moves the fraction less. Before: 40 correct and 10 wrong gives fraction 4/5. After: 39 correct and 11 wrong gives fraction 39/50.

4050=4/5→3950=39/50{40\over50}=4/5\quad\to\quad {39\over50}=39/50
One row moves a small metricExact accuracy movement by held-out slice size.one row metric moveslicerowsbefore correct/wrongbefore accafter correct/wrongafter accmovetiny held-out slice54/14/53/23/51/5larger held-out slice5040/104/539/1139/501/50one correct row changes to wrong in each slicearithmetic sensitivity to sample size only; one pinned held-out slice; does not prove future performance; NOT training behavior; NOTuncertainty intervals; NOT generalizationthreshold fixed first; metric alone is not a claim; NOT training

Compare the moves

The tiny slice moves by 1/5. The larger slice moves by 1/50. This shows arithmetic sensitivity to sample size only; it does not prove future performance, training behavior, uncertainty intervals, or generalization.

∣4/5−3/5∣=1/5,∣4/5−39/50∣=1/50\left|4/5 - 3/5\right|=1/5,\quad \left|4/5 - 39/50\right|=1/50
One row moves a small metricExact accuracy movement by held-out slice size.one row metric moveslicerowsbefore correct/wrongbefore accafter correct/wrongafter accmovetiny held-out slice54/14/53/23/51/5larger held-out slice5040/104/539/1139/501/50one correct row changes to wrong in each slicearithmetic sensitivity to sample size only; one pinned held-out slice; does not prove future performance; NOT training behavior; NOTuncertainty intervals; NOT generalizationthreshold fixed first; metric alone is not a claim; NOT training