Surprise Bits

One Small Probability Raises the Mean

One smaller assigned true-label probability can add more exact surprise bits, raising the mean without any decimal loss values.

highlighted = computed this step

Show four assigned probabilities

Use 4 displayed rows. The first three true-label probabilities are 1/2, and the fourth is 1/32. These are assigned toy probabilities for loss arithmetic.

ptrue=(1/2,1/2,1/2,1/32)p_{\text{true}}=(1/2,1/2,1/2,1/32)
One small probability raises the meanExact surprise bits when one assigned true-label probability is small.one small probability raises the meanrowtrue-label psurprise bitsbits barrunning sumrow 11/21#1row 21/21#2row 31/21#3row 41/325#####8sum=8 bitsmean=2 bits over 4 rowsassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT aprobability calibration claimassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOTtraining; NOT generalization; NOT a probability calibration claim

Three rows add one bit each

Each 1/2 row contributes 1 bit, so the first 3 rows have running sum 3 bits.

1+1+1=31 + 1 + 1 = 3
One small probability raises the meanExact surprise bits when one assigned true-label probability is small.one small probability raises the meanrowtrue-label psurprise bitsbits barrunning sumrow 11/21#1row 21/21#2row 31/21#3row 41/325#####8sum=8 bitsmean=2 bits over 4 rowsassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT aprobability calibration claimassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOTtraining; NOT generalization; NOT a probability calibration claim

The small probability adds five bits

The fourth true-label probability is 1/32. With log base 2, that row contributes 5 bits, taking the sum to 8 bits.

−log⁡2(1/32)=5,3+5=8-\log_{2}(1/32)=5,\quad 3 + 5 = 8
One small probability raises the meanExact surprise bits when one assigned true-label probability is small.one small probability raises the meanrowtrue-label psurprise bitsbits barrunning sumrow 11/21#1row 21/21#2row 31/21#3row 41/325#####8sum=8 bitsmean=2 bits over 4 rowsassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT aprobability calibration claimassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOTtraining; NOT generalization; NOT a probability calibration claim

Mean over the four rows

The mean is 8 bits divided by 4 rows, which is 2 bits. The fourth row is not a verdict on training, generalization, or calibration; it is assigned loss arithmetic.

84=2{8\over4}=2
One small probability raises the meanExact surprise bits when one assigned true-label probability is small.one small probability raises the meanrowtrue-label psurprise bitsbits barrunning sumrow 11/21#1row 21/21#2row 31/21#3row 41/325#####8sum=8 bitsmean=2 bits over 4 rowsassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT aprobability calibration claimassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOTtraining; NOT generalization; NOT a probability calibration claim