Validation rows can choose a threshold before the held-out test rows are measured. This lesson freezes one exact threshold and then counts the test grid once.
State the validation rule
The validation rows try thresholds 5, 3, and 1. The rule is pinned before the test rows are read: choose the highest threshold whose validation recall is at least 2/3.
choose highest t with R val ≥ 2 / 3 \text{choose highest }t\text{ with }R_{\text{val}}\ge 2/3 choose highest t with R val ≥ 2/3 Validation chooses the threshold Exact validation choice and frozen test measurement. validation chooses threshold threshold precision recall rule check 5 1 1/3 skip 3 2/3 2/3 choose 1 3/5 1 skip choose the highest threshold with validation recall at least 2/3; chosen threshold=3 frozen threshold on held-out test rows row actual score output cell t1 positive 4 positive TP t2 positive 2 negative FN t3 negative 5 positive FP t4 negative 1 negative TN threshold frozen at 3 test confusion grid, measured once output positive negative actual positive negative TP 1 FN 1 FP 1 TN 1 accuracy=1/2 precision=1/2, recall=1/2 validation chooses; test measures once; retuning after seeing test rows breaks the clean test result; does not prove future performance; NOT training; NOT generalization threshold fixed first; metric alone is not a claim; NOT training
Choose from validation only
Threshold 5 has recall 1/3, so it misses the rule. Threshold 3 has recall 2/3 and is the highest threshold that passes. Threshold 1 also passes, but it is lower.
t = 3 chosen on validation rows t=3\quad \text{chosen on validation rows} t = 3 chosen on validation rows Validation chooses the threshold Exact validation choice and frozen test measurement. validation chooses threshold threshold precision recall rule check 5 1 1/3 skip 3 2/3 2/3 choose 1 3/5 1 skip choose the highest threshold with validation recall at least 2/3; chosen threshold=3 frozen threshold on held-out test rows row actual score output cell t1 positive 4 positive TP t2 positive 2 negative FN t3 negative 5 positive FP t4 negative 1 negative TN threshold frozen at 3 test confusion grid, measured once output positive negative actual positive negative TP 1 FN 1 FP 1 TN 1 accuracy=1/2 precision=1/2, recall=1/2 validation chooses; test measures once; retuning after seeing test rows breaks the clean test result; does not prove future performance; NOT training; NOT generalization threshold fixed first; metric alone is not a claim; NOT training
Freeze and count test rows
Now freeze threshold 3 and count the held-out test rows once. The four cells are TP=1, FN=1, FP=1, and TN=1.
T P = 1 , F N = 1 , F P = 1 , T N = 1 TP=1,\;FN=1,\;FP=1,\;TN=1 TP = 1 , FN = 1 , FP = 1 , TN = 1 Validation chooses the threshold Exact validation choice and frozen test measurement. validation chooses threshold threshold precision recall rule check 5 1 1/3 skip 3 2/3 2/3 choose 1 3/5 1 skip choose the highest threshold with validation recall at least 2/3; chosen threshold=3 frozen threshold on held-out test rows row actual score output cell t1 positive 4 positive TP t2 positive 2 negative FN t3 negative 5 positive FP t4 negative 1 negative TN threshold frozen at 3 test confusion grid, measured once output positive negative actual positive negative TP 1 FN 1 FP 1 TN 1 accuracy=1/2 precision=1/2, recall=1/2 validation chooses; test measures once; retuning after seeing test rows breaks the clean test result; does not prove future performance; NOT training; NOT generalization threshold fixed first; metric alone is not a claim; NOT training
Boundary
On those held-out rows, accuracy, precision, and recall are all 1/2. Validation chooses; test measures once. Retuning after seeing test rows would no longer be a clean test result, and this does not prove future performance or training behavior.
accuracy = precision = recall = 1 / 2 \operatorname{accuracy}=\operatorname{precision}=\operatorname{recall}=1/2 accuracy = precision = recall = 1/2 Validation chooses the threshold Exact validation choice and frozen test measurement. validation chooses threshold threshold precision recall rule check 5 1 1/3 skip 3 2/3 2/3 choose 1 3/5 1 skip choose the highest threshold with validation recall at least 2/3; chosen threshold=3 frozen threshold on held-out test rows row actual score output cell t1 positive 4 positive TP t2 positive 2 negative FN t3 negative 5 positive FP t4 negative 1 negative TN threshold frozen at 3 test confusion grid, measured once output positive negative actual positive negative TP 1 FN 1 FP 1 TN 1 accuracy=1/2 precision=1/2, recall=1/2 validation chooses; test measures once; retuning after seeing test rows breaks the clean test result; does not prove future performance; NOT training; NOT generalization threshold fixed first; metric alone is not a claim; NOT training