Held-Out Rows

Freeze the Baseline

After the train-only mean is computed, keep it fixed for held-out scoring.

highlighted = computed this step

Freeze the constant baseline

The constant baseline is the train mean: 4. That value is now used as the prediction for held-out rows.

y^constant=4\hat y_{\text{constant}}=4
Train Test Split ExactlyExact train-only statistic and held-out scoring rows.train-only mean and held-out scoringrowroleytrain stat?predictionabs errorAtrain2yes--Btrain4yes--Ctrain6yes--Dtest3held out41Etest7held out43train mean: (2+4+6)/3 = 4test MAE: (1+3)/2 = 2train statistic uses train rows onlytest rows are held out and scored onceNOT training; NOT learning; NOT generalizationNOT future performance; NOT unbiased; NOT safeNOT best split; NOT an accuracy guarantee

Both held-out predictions use the same value

For row D, prediction is 4. For row E, prediction is also 4.

y^D=4,y^E=4\hat y_D=4,\quad \hat y_E=4
Train Test Split ExactlyExact train-only statistic and held-out scoring rows.train-only mean and held-out scoringrowroleytrain stat?predictionabs errorAtrain2yes--Btrain4yes--Ctrain6yes--Dtest3held out41Etest7held out43train mean: (2+4+6)/3 = 4test MAE: (1+3)/2 = 2train statistic uses train rows onlytest rows are held out and scored onceNOT training; NOT learning; NOT generalizationNOT future performance; NOT unbiased; NOT safeNOT best split; NOT an accuracy guarantee

Do not retune on the test rows

The test rows are scored with the frozen baseline. They do not change the train mean after being seen.

freeze, then score held-out rows\text{freeze, then score held-out rows}
Train Test Split ExactlyExact train-only statistic and held-out scoring rows.train-only mean and held-out scoringrowroleytrain stat?predictionabs errorAtrain2yes--Btrain4yes--Ctrain6yes--Dtest3held out41Etest7held out43train mean: (2+4+6)/3 = 4test MAE: (1+3)/2 = 2train statistic uses train rows onlytest rows are held out and scored onceNOT training; NOT learning; NOT generalizationNOT future performance; NOT unbiased; NOT safeNOT best split; NOT an accuracy guarantee