Evaluation Metrics

Stratified Split Keeps Class Mix Visible

A split can change the visible class mix before any validation or test metric is read. This lesson compares one drifting split with one stratified split using exact row counts.

highlighted = computed this step

Start with the class mix

The toy dataset has 8 rows: 2 positive and 6 negative. Each shown split uses 4 train rows and 4 test rows.

N=8,+=2,−=6N=8,\quad +=2,\quad -=6
Stratified split keeps class mix visibleExact train/test class mix comparison.class mix in train/test splitssplittrain +train -train + sharetest +test -test + sharevisible resultunstratified example040221/2mix drift visiblestratified example131/4131/4same 1/4 positive sharedataset: 2 positive + 6 negative = 8 rowsstratified row keeps train and test at 1/4 positive sharepreserves the toy label mix; not a guarantee for future rows; NOT training; NOT generalizationthreshold fixed first; metric alone is not a claim; NOT training

Unstratified split can drift

In the unstratified example, train has 0 positive out of 4 rows, while test has 2 positives out of 4 rows. The visible shares are 0 and 1/2.

train +=04=0,test +=24=1/2\text{train }+={0\over4}=0,\quad \text{test }+={2\over4}=1/2
Stratified split keeps class mix visibleExact train/test class mix comparison.class mix in train/test splitssplittrain +train -train + sharetest +test -test + sharevisible resultunstratified example040221/2mix drift visiblestratified example131/4131/4same 1/4 positive sharedataset: 2 positive + 6 negative = 8 rowsstratified row keeps train and test at 1/4 positive sharepreserves the toy label mix; not a guarantee for future rows; NOT training; NOT generalizationthreshold fixed first; metric alone is not a claim; NOT training

Stratified split keeps the shown mix

In the stratified example, train has 1 positive out of 4 rows, and test also has 1 positive out of 4 rows. Both sides show positive share 1/4.

train +=14=1/4,test +=14=1/4\text{train }+={1\over4}=1/4,\quad \text{test }+={1\over4}=1/4
Stratified split keeps class mix visibleExact train/test class mix comparison.class mix in train/test splitssplittrain +train -train + sharetest +test -test + sharevisible resultunstratified example040221/2mix drift visiblestratified example131/4131/4same 1/4 positive sharedataset: 2 positive + 6 negative = 8 rowsstratified row keeps train and test at 1/4 positive sharepreserves the toy label mix; not a guarantee for future rows; NOT training; NOT generalizationthreshold fixed first; metric alone is not a claim; NOT training

Boundary

This preserves the toy label mix before validation or test metrics are read. It is not a guarantee for future rows, NOT training, and NOT generalization.

same shown mix≠future guarantee\text{same shown mix}\ne\text{future guarantee}
Stratified split keeps class mix visibleExact train/test class mix comparison.class mix in train/test splitssplittrain +train -train + sharetest +test -test + sharevisible resultunstratified example040221/2mix drift visiblestratified example131/4131/4same 1/4 positive sharedataset: 2 positive + 6 negative = 8 rowsstratified row keeps train and test at 1/4 positive sharepreserves the toy label mix; not a guarantee for future rows; NOT training; NOT generalizationthreshold fixed first; metric alone is not a claim; NOT training