Two toy models can have the same visible correct count while assigned true-label probabilities give different exact bit losses.
Both toy models show the same correct count
Model A and Model B both have correct count 2/2 in this assigned toy comparison. That count is visible before looking at the loss bits.
Model A correct = 2 / 2 , Model B correct = 2 / 2 \text{Model A correct}=2/2,\quad \text{Model B correct}=2/2 Model A correct = 2/2 , Model B correct = 2/2 Same correct count can have different loss Exact bit loss changes when assigned true-label probabilities change. same correct count, different loss model correct count true-label p bits sum mean Model A 2/2 [1/2, 1/2] [1, 1] 2 1 Model B 2/2 [1/2, 1/8] [1, 3] 4 2 same correct count 2/2; different exact loss correct-count is assigned for this toy comparison; assigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT a probability calibration claim; NOT calibration proof correct-count is assigned for this toy comparison; assigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT a probability calibration claim; NOT calibration proof
Model A has two one-bit rows
Model A assigns true-label probabilities 1/2 and 1/2. Those are 1 bit and 1 bit, so the sum is 2 and the mean is 1.
1 + 1 = 2 , 2 / 2 = 1 1 + 1 = 2,\quad 2/2=1 1 + 1 = 2 , 2/2 = 1 Same correct count can have different loss Exact bit loss changes when assigned true-label probabilities change. same correct count, different loss model correct count true-label p bits sum mean Model A 2/2 [1/2, 1/2] [1, 1] 2 1 Model B 2/2 [1/2, 1/8] [1, 3] 4 2 same correct count 2/2; different exact loss correct-count is assigned for this toy comparison; assigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT a probability calibration claim; NOT calibration proof correct-count is assigned for this toy comparison; assigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT a probability calibration claim; NOT calibration proof
Model B has one larger bit row
Model B assigns true-label probabilities 1/2 and 1/8. Those are 1 bit and 3 bits, so the sum is 4 and the mean is 2.
1 + 3 = 4 , 4 / 2 = 2 1 + 3 = 4,\quad 4/2=2 1 + 3 = 4 , 4/2 = 2 Same correct count can have different loss Exact bit loss changes when assigned true-label probabilities change. same correct count, different loss model correct count true-label p bits sum mean Model A 2/2 [1/2, 1/2] [1, 1] 2 1 Model B 2/2 [1/2, 1/8] [1, 3] 4 2 same correct count 2/2; different exact loss correct-count is assigned for this toy comparison; assigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT a probability calibration claim; NOT calibration proof correct-count is assigned for this toy comparison; assigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT a probability calibration claim; NOT calibration proof
The count matched, but the loss did not
The visible correct count is the same, but the exact mean losses differ: 1 bit for Model A and 2 bits for Model B. This is assigned loss arithmetic only. No claim about training, generalization, or calibration is made.
Same correct count can have different loss Exact bit loss changes when assigned true-label probabilities change. same correct count, different loss model correct count true-label p bits sum mean Model A 2/2 [1/2, 1/2] [1, 1] 2 1 Model B 2/2 [1/2, 1/8] [1, 3] 4 2 same correct count 2/2; different exact loss correct-count is assigned for this toy comparison; assigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT a probability calibration claim; NOT calibration proof correct-count is assigned for this toy comparison; assigned toy probabilities; log base 2 chosen for exact bits; loss arithmetic; NOT training; NOT generalization; NOT a probability calibration claim; NOT calibration proof