Evaluation, Honestly
A lightweight execution-visualization book.
Evaluation Metrics
- The Confusion Grid
- Accuracy Can Hide Misses
- Balanced Accuracy Keeps Both Sides
- False-Positive Rate Uses Actual Negatives
- Overall Fraction Can Hide a Slice
- Train/Test Split and Leakage
- Stratified Split Keeps Class Mix Visible
- Validation Chooses the Threshold
- One Row Moves a Small Metric
- Moving the Threshold
- Precision-Recall Trade-off
- F1 Is a Summary
- Mistake Costs Change the Choice
- Base Rate Changes Precision
- Calibration Needs Buckets
- Metric Is Not Generalization