Close the boundary around one exact split and one exact held-out score.

highlighted = computed this step

The no-leakage boundary

The train statistic uses train rows only. The test rows are held out and scored once.

train first; score test once\text{train first; score test once}
Train Test Split ExactlyExact train-only statistic and held-out scoring rows.train-only mean and held-out scoringrowroleytrain stat?predictionabs errorAtrain2yes--Btrain4yes--Ctrain6yes--Dtest3held out41Etest7held out43train mean: (2+4+6)/3 = 4test MAE: (1+3)/2 = 2train statistic uses train rows onlytest rows are held out and scored onceNOT training; NOT learning; NOT generalizationNOT future performance; NOT unbiased; NOT safeNOT best split; NOT an accuracy guarantee

What this does not promise

This split does not prove generalization, future performance, unbiased measurement, safety, or the best split.

split arithmetic only\text{split arithmetic only}
Train Test Split ExactlyExact train-only statistic and held-out scoring rows.train-only mean and held-out scoringrowroleytrain stat?predictionabs errorAtrain2yes--Btrain4yes--Ctrain6yes--Dtest3held out41Etest7held out43train mean: (2+4+6)/3 = 4test MAE: (1+3)/2 = 2train statistic uses train rows onlytest rows are held out and scored onceNOT training; NOT learning; NOT generalizationNOT future performance; NOT unbiased; NOT safeNOT best split; NOT an accuracy guarantee

The honest close

This book pins one split and one constant baseline. It is NOT training, NOT learning, NOT generalization, NOT future performance, NOT unbiased, NOT safe, NOT best split, and NOT an accuracy guarantee.

held-out rows stay held out\text{held-out rows stay held out}
Train Test Split ExactlyExact train-only statistic and held-out scoring rows.train-only mean and held-out scoringrowroleytrain stat?predictionabs errorAtrain2yes--Btrain4yes--Ctrain6yes--Dtest3held out41Etest7held out43train mean: (2+4+6)/3 = 4test MAE: (1+3)/2 = 2train statistic uses train rows onlytest rows are held out and scored onceNOT training; NOT learning; NOT generalizationNOT future performance; NOT unbiased; NOT safeNOT best split; NOT an accuracy guarantee