Buckets and Observed Fractions

Reliability Table Compares Fractions

Place bucket score and observed fraction side by side.

highlighted = computed this step

Put score and observed side by side

The reliability table compares each bucket score to its observed positive fraction.

score column beside observed column\text{score column beside observed column}
Calibration Bins ExactlyExact observed fractions inside pinned score buckets.score buckets with exact observed fractionshigh bucketscore=3/4Alabel 1Blabel 1Clabel 1Dlabel 0positives=3rows=4observed=3/4 = 3/4score matches observedlow bucketscore=1/4Elabel 1Flabel 0Glabel 0Hlabel 0positives=1rows=4observed=1/4 = 1/4score matches observedcontrast high bucketscore=3/4Ilabel 1Jlabel 1Klabel 0Llabel 0positives=2rows=4observed=2/4 = 1/2mismatch: 3/4 -> 1/2reliability table: score vs observed fractionbucketscoreobservedstatushigh bucket3/43/4same fractionlow bucket1/41/4same fractioncontrast high bucket3/41/2visible mismatchObserved fraction is positives over rows inside one displayed bucketcontrast is one displayed mismatch onlydoes not claim: NOT probability truth; NOT future calibrationNOT accuracy; NOT fairness; NOT safety; NOT generalizationNOT best model; NOT future performance

High row in the table

For the high bucket, score 3/4 and observed fraction 3/4 are the same.

3/4=3/43/4 = 3/4
Calibration Bins ExactlyExact observed fractions inside pinned score buckets.score buckets with exact observed fractionshigh bucketscore=3/4Alabel 1Blabel 1Clabel 1Dlabel 0positives=3rows=4observed=3/4 = 3/4score matches observedlow bucketscore=1/4Elabel 1Flabel 0Glabel 0Hlabel 0positives=1rows=4observed=1/4 = 1/4score matches observedcontrast high bucketscore=3/4Ilabel 1Jlabel 1Klabel 0Llabel 0positives=2rows=4observed=2/4 = 1/2mismatch: 3/4 -> 1/2reliability table: score vs observed fractionbucketscoreobservedstatushigh bucket3/43/4same fractionlow bucket1/41/4same fractioncontrast high bucket3/41/2visible mismatchObserved fraction is positives over rows inside one displayed bucketcontrast is one displayed mismatch onlydoes not claim: NOT probability truth; NOT future calibrationNOT accuracy; NOT fairness; NOT safety; NOT generalizationNOT best model; NOT future performance

Low row in the table

For the low bucket, score 1/4 and observed fraction 1/4 are the same.

1/4=1/41/4 = 1/4
Calibration Bins ExactlyExact observed fractions inside pinned score buckets.score buckets with exact observed fractionshigh bucketscore=3/4Alabel 1Blabel 1Clabel 1Dlabel 0positives=3rows=4observed=3/4 = 3/4score matches observedlow bucketscore=1/4Elabel 1Flabel 0Glabel 0Hlabel 0positives=1rows=4observed=1/4 = 1/4score matches observedcontrast high bucketscore=3/4Ilabel 1Jlabel 1Klabel 0Llabel 0positives=2rows=4observed=2/4 = 1/2mismatch: 3/4 -> 1/2reliability table: score vs observed fractionbucketscoreobservedstatushigh bucket3/43/4same fractionlow bucket1/41/4same fractioncontrast high bucket3/41/2visible mismatchObserved fraction is positives over rows inside one displayed bucketcontrast is one displayed mismatch onlydoes not claim: NOT probability truth; NOT future calibrationNOT accuracy; NOT fairness; NOT safety; NOT generalizationNOT best model; NOT future performance