Surprise Bits

Surprise Adds Over Rows

Cross-entropy can be read as surprise added across rows. This first slice uses powers of two so every displayed bit value is exact.

highlighted = computed this step

Use assigned toy probabilities

Start with 3 rows. The true-label probabilities are 1/2, 1/4, and 1/2. They are assigned toy probabilities, not calibration evidence.

ptrue=(1/2,1/4,1/2)p_{\text{true}}=(1/2,1/4,1/2)
Cross-entropy honestlyExact surprise bits for assigned powers-of-two probabilities.surprise bits add over rowsrowtrue-label psurprise bitsbits barrunning sumrow 11/21#1row 21/42##3row 31/21#4sum=4 bitsmean=4/3 bits over 3 rowsassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT aprobability calibration claimassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOTtraining; NOT generalization; NOT a probability calibration claim

Read exact surprise bits

With log base 2, probability 1/2 has surprise 1 bit, and probability 1/4 has surprise 2 bits.

−log⁡2(1/2)=1,−log⁡2(1/4)=2-\log_{2}(1/2)=1,\quad -\log_{2}(1/4)=2
Cross-entropy honestlyExact surprise bits for assigned powers-of-two probabilities.surprise bits add over rowsrowtrue-label psurprise bitsbits barrunning sumrow 11/21#1row 21/42##3row 31/21#4sum=4 bitsmean=4/3 bits over 3 rowsassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT aprobability calibration claimassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOTtraining; NOT generalization; NOT a probability calibration claim

Surprise adds over rows

The row surprises are 1, 2, and 1 bits. The running sum ends at 4 bits.

1+2+1=41 + 2 + 1 = 4
Cross-entropy honestlyExact surprise bits for assigned powers-of-two probabilities.surprise bits add over rowsrowtrue-label psurprise bitsbits barrunning sumrow 11/21#1row 21/42##3row 31/21#4sum=4 bitsmean=4/3 bits over 3 rowsassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT aprobability calibration claimassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOTtraining; NOT generalization; NOT a probability calibration claim

Mean over the rows

The mean loss is the sum divided by the 3 rows: 4/3 bits. This is loss arithmetic only: NOT training, NOT generalization, and NOT a probability calibration claim.

43=4/3{4\over3}=4/3
Cross-entropy honestlyExact surprise bits for assigned powers-of-two probabilities.surprise bits add over rowsrowtrue-label psurprise bitsbits barrunning sumrow 11/21#1row 21/42##3row 31/21#4sum=4 bitsmean=4/3 bits over 3 rowsassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT aprobability calibration claimassigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOTtraining; NOT generalization; NOT a probability calibration claim