Evaluation Metrics

Calibration Needs Buckets

Calibration checks compare displayed scores with observed positive fractions inside score buckets. This lesson keeps both bucket fractions exact.

highlighted = computed this step

Bucket A

Bucket A groups rows with displayed score 3/4. It has 4 rows: 2 positives and 2 negatives. The observed positive fraction is 1/2, so it does not match the score.

score=3/4,observed=24=1/2\text{score}=3/4,\quad \text{observed}={2\over4}=1/2
Calibration needs bucketsExact bucket fractions for shown scores.calibration bucketsbucketscorerowspositivenegativeobserved +gapcheckBucket A3/44221/21/4does not matchBucket B1/44131/40matches exactlyobserved fraction = positives / rows; gap = |score - observed|one tiny held-out slice; does not prove future calibration; NOT training; NOT generalization; NOT an overall model-quality claimthreshold fixed first; metric alone is not a claim; NOT training

Bucket B

Bucket B groups rows with displayed score 1/4. It has 4 rows: 1 positive and 3 negatives. The observed positive fraction is 1/4, so this bucket matches exactly.

score=1/4,observed=14=1/4\text{score}=1/4,\quad \text{observed}={1\over4}=1/4
Calibration needs bucketsExact bucket fractions for shown scores.calibration bucketsbucketscorerowspositivenegativeobserved +gapcheckBucket A3/44221/21/4does not matchBucket B1/44131/40matches exactlyobserved fraction = positives / rows; gap = |score - observed|one tiny held-out slice; does not prove future calibration; NOT training; NOT generalization; NOT an overall model-quality claimthreshold fixed first; metric alone is not a claim; NOT training

Compare the gaps

Bucket A has gap 1/4. Bucket B has gap 0. This is one tiny held-out slice; it does not prove future calibration, training quality, or generalization.

∣3/4−1/2∣=1/4,∣1/4−1/4∣=0\left|3/4 - 1/2\right|=1/4,\quad \left|1/4 - 1/4\right|=0
Calibration needs bucketsExact bucket fractions for shown scores.calibration bucketsbucketscorerowspositivenegativeobserved +gapcheckBucket A3/44221/21/4does not matchBucket B1/44131/40matches exactlyobserved fraction = positives / rows; gap = |score - observed|one tiny held-out slice; does not prove future calibration; NOT training; NOT generalization; NOT an overall model-quality claimthreshold fixed first; metric alone is not a claim; NOT training