Start with exact score buckets and visible row labels.

highlighted = computed this step

Start with bucket scores

The high bucket carries score 3/4. The low bucket carries score 1/4.

bucket scores 3/4 and 1/4\text{bucket scores }3/4\text{ and }1/4
Calibration Bins ExactlyExact observed fractions inside pinned score buckets.score buckets with exact observed fractionshigh bucketscore=3/4Alabel 1Blabel 1Clabel 1Dlabel 0positives=3rows=4observed=3/4 = 3/4score matches observedlow bucketscore=1/4Elabel 1Flabel 0Glabel 0Hlabel 0positives=1rows=4observed=1/4 = 1/4score matches observedcontrast high bucketscore=3/4Ilabel 1Jlabel 1Klabel 0Llabel 0positives=2rows=4observed=2/4 = 1/2mismatch: 3/4 -> 1/2reliability table: score vs observed fractionbucketscoreobservedstatushigh bucket3/43/4same fractionlow bucket1/41/4same fractioncontrast high bucket3/41/2visible mismatchObserved fraction is positives over rows inside one displayed bucketcontrast is one displayed mismatch onlydoes not claim: NOT probability truth; NOT future calibrationNOT accuracy; NOT fairness; NOT safety; NOT generalizationNOT best model; NOT future performance

Each bucket holds rows

The high bucket holds rows A, B, C, D. The low bucket holds rows E, F, G, H.

rows stay inside their displayed bucket\text{rows stay inside their displayed bucket}
Calibration Bins ExactlyExact observed fractions inside pinned score buckets.score buckets with exact observed fractionshigh bucketscore=3/4Alabel 1Blabel 1Clabel 1Dlabel 0positives=3rows=4observed=3/4 = 3/4score matches observedlow bucketscore=1/4Elabel 1Flabel 0Glabel 0Hlabel 0positives=1rows=4observed=1/4 = 1/4score matches observedcontrast high bucketscore=3/4Ilabel 1Jlabel 1Klabel 0Llabel 0positives=2rows=4observed=2/4 = 1/2mismatch: 3/4 -> 1/2reliability table: score vs observed fractionbucketscoreobservedstatushigh bucket3/43/4same fractionlow bucket1/41/4same fractioncontrast high bucket3/41/2visible mismatchObserved fraction is positives over rows inside one displayed bucketcontrast is one displayed mismatch onlydoes not claim: NOT probability truth; NOT future calibrationNOT accuracy; NOT fairness; NOT safety; NOT generalizationNOT best model; NOT future performance

Labels stay visible

Every row shows a pinned label, either 1 or 0.

visible labels only\text{visible labels only}
Calibration Bins ExactlyExact observed fractions inside pinned score buckets.score buckets with exact observed fractionshigh bucketscore=3/4Alabel 1Blabel 1Clabel 1Dlabel 0positives=3rows=4observed=3/4 = 3/4score matches observedlow bucketscore=1/4Elabel 1Flabel 0Glabel 0Hlabel 0positives=1rows=4observed=1/4 = 1/4score matches observedcontrast high bucketscore=3/4Ilabel 1Jlabel 1Klabel 0Llabel 0positives=2rows=4observed=2/4 = 1/2mismatch: 3/4 -> 1/2reliability table: score vs observed fractionbucketscoreobservedstatushigh bucket3/43/4same fractionlow bucket1/41/4same fractioncontrast high bucket3/41/2visible mismatchObserved fraction is positives over rows inside one displayed bucketcontrast is one displayed mismatch onlydoes not claim: NOT probability truth; NOT future calibrationNOT accuracy; NOT fairness; NOT safety; NOT generalizationNOT best model; NOT future performance