Close the boundary around exact bucket arithmetic.
Bins are local summaries
Each observed fraction is computed inside one displayed bucket.
positives over rows in that bucket \text{positives over rows in that bucket} positives over rows in that bucket Calibration Bins Exactly Exact observed fractions inside pinned score buckets. score buckets with exact observed fractions high bucket score=3/4 A label 1 B label 1 C label 1 D label 0 positives=3 rows=4 observed=3/4 = 3/4 score matches observed low bucket score=1/4 E label 1 F label 0 G label 0 H label 0 positives=1 rows=4 observed=1/4 = 1/4 score matches observed contrast high bucket score=3/4 I label 1 J label 1 K label 0 L label 0 positives=2 rows=4 observed=2/4 = 1/2 mismatch: 3/4 -> 1/2 reliability table: score vs observed fraction bucket score observed status high bucket 3/4 3/4 same fraction low bucket 1/4 1/4 same fraction contrast high bucket 3/4 1/2 visible mismatch Observed fraction is positives over rows inside one displayed bucket contrast is one displayed mismatch only does not claim: NOT probability truth; NOT future calibration NOT accuracy; NOT fairness; NOT safety; NOT generalization NOT best model; NOT future performance
A mismatch is still a displayed slice
The mismatch arrow is a visible comparison for one contrast bucket, not a global verdict.
one displayed mismatch only \text{one displayed mismatch only} one displayed mismatch only Calibration Bins Exactly Exact observed fractions inside pinned score buckets. score buckets with exact observed fractions high bucket score=3/4 A label 1 B label 1 C label 1 D label 0 positives=3 rows=4 observed=3/4 = 3/4 score matches observed low bucket score=1/4 E label 1 F label 0 G label 0 H label 0 positives=1 rows=4 observed=1/4 = 1/4 score matches observed contrast high bucket score=3/4 I label 1 J label 1 K label 0 L label 0 positives=2 rows=4 observed=2/4 = 1/2 mismatch: 3/4 -> 1/2 reliability table: score vs observed fraction bucket score observed status high bucket 3/4 3/4 same fraction low bucket 1/4 1/4 same fraction contrast high bucket 3/4 1/2 visible mismatch Observed fraction is positives over rows inside one displayed bucket contrast is one displayed mismatch only does not claim: NOT probability truth; NOT future calibration NOT accuracy; NOT fairness; NOT safety; NOT generalization NOT best model; NOT future performance
The honest boundary
This is exact bucket arithmetic only. It does not claim: NOT probability truth; NOT future calibration; NOT accuracy; NOT fairness; NOT safety; NOT generalization; NOT best model; NOT future performance.
exact bucket counts; exact observed fractions \text{exact bucket counts; exact observed fractions} exact bucket counts; exact observed fractions Calibration Bins Exactly Exact observed fractions inside pinned score buckets. score buckets with exact observed fractions high bucket score=3/4 A label 1 B label 1 C label 1 D label 0 positives=3 rows=4 observed=3/4 = 3/4 score matches observed low bucket score=1/4 E label 1 F label 0 G label 0 H label 0 positives=1 rows=4 observed=1/4 = 1/4 score matches observed contrast high bucket score=3/4 I label 1 J label 1 K label 0 L label 0 positives=2 rows=4 observed=2/4 = 1/2 mismatch: 3/4 -> 1/2 reliability table: score vs observed fraction bucket score observed status high bucket 3/4 3/4 same fraction low bucket 1/4 1/4 same fraction contrast high bucket 3/4 1/2 visible mismatch Observed fraction is positives over rows inside one displayed bucket contrast is one displayed mismatch only does not claim: NOT probability truth; NOT future calibration NOT accuracy; NOT fairness; NOT safety; NOT generalization NOT best model; NOT future performance