Cross-Entropy, Honestly

Surprise Bits

• Surprise Adds Over Rows
• One Small Probability Raises the Mean
• The Average Hides the Row
• Same Correct Count Can Have Different Loss