Logistic Classification

Log Loss (Small Example)

Compute log loss = −(1/n)Σ(yᵢ·ln(pᵢ) + (1−yᵢ)·ln(1−pᵢ)) for true binary labels y and predicted probabilities p. A loop applies math.log per sample. Library: sklearn.metrics.log_loss(y_true, p_pred, labels=[0,1]). RESULT: log loss (rounded).

By hand

y=[1,0,1,0,1], p=[0.9,0.2,0.7,0.3,0.8]. Per-sample terms: −ln(0.9)=0.1054, −ln(0.8)=0.2231, −ln(0.7)=0.3567, −ln(0.7)=0.3567, −ln(0.8)=0.2231. Sum=1.265, loss=1.265/5=0.253.

naive.py
Replay: real traced execution (multi-file project)
import math
y_true = [1, 0, 1, 0, 1]
p_pred = [0.9, 0.2, 0.7, 0.3, 0.8]
n = len(y_true)
total = 0.0
for i in range(n):
    yi = y_true[i]
    pi = p_pred[i]
    total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))
loss = total / n
print('RESULT:', round(loss, 4))

  1. import math

    1import math2y_true = [1, 0, 1, 0, 1]

  2. y_true ← [1, 0, 1, 0, 1]

    1import math2y_true = [1, 0, 1, 0, 1]3p_pred = [0.9, 0.2, 0.7, 0.3, 0.8]
    values this step[1, 0, 1, 0, 1]y_true

  3. p_pred ← [0.9, 0.2, 0.7, 0.3, 0.8]

    2y_true = [1, 0, 1, 0, 1]3p_pred = [0.9, 0.2, 0.7, 0.3, 0.8]4n = len(y_true)
    values this step[0.9, 0.2, 0.7, 0.3, 0.8]p_pred

  4. n ← 5

    3p_pred = [0.9, 0.2, 0.7, 0.3, 0.8]4n = len(y_true)5total = 0.0
    values this step5n

  5. total ← 0.0

    4n = len(y_true)5total = 0.06for i in range(n):
    values this step0.0total

  6. i ← 0

    5total = 0.06for i in range(n):7    yi = y_true[i]
    values this step0i

  7. yi ← 1

    6for i in range(n):7    yi = y_true[i]8    pi = p_pred[i]
    values this step1yi

  8. pi ← 0.9

    7yi = y_true[i]8pi = p_pred[i]9total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))
    values this step0.9pi

  9. total ← 0.10536051565782628

    8    pi = p_pred[i]9    total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))10loss = total / n
    values this step0.0 → 0.10536051565782628total

  10. i ← 1

    5total = 0.06for i in range(n):7    yi = y_true[i]
    values this step0 → 1i

  11. yi ← 0

    6for i in range(n):7    yi = y_true[i]8    pi = p_pred[i]
    values this step1 → 0yi

  12. pi ← 0.2

    7yi = y_true[i]8pi = p_pred[i]9total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))
    values this step0.9 → 0.2pi

  13. total ← 0.328504066972036

    8    pi = p_pred[i]9    total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))10loss = total / n
    values this step0.10536051565782628 → 0.328504066972036total

  14. i ← 2

    5total = 0.06for i in range(n):7    yi = y_true[i]
    values this step1 → 2i

  15. yi ← 1

    6for i in range(n):7    yi = y_true[i]8    pi = p_pred[i]
    values this step0 → 1yi

  16. pi ← 0.7

    7yi = y_true[i]8pi = p_pred[i]9total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))
    values this step0.2 → 0.7pi

  17. total ← 0.6851790109107685

    8    pi = p_pred[i]9    total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))10loss = total / n
    values this step0.328504066972036 → 0.6851790109107685total

  18. i ← 3

    5total = 0.06for i in range(n):7    yi = y_true[i]
    values this step2 → 3i

  19. yi ← 0

    6for i in range(n):7    yi = y_true[i]8    pi = p_pred[i]
    values this step1 → 0yi

  20. pi ← 0.3

    7yi = y_true[i]8pi = p_pred[i]9total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))
    values this step0.7 → 0.3pi

  21. total ← 1.041853954849501

    8    pi = p_pred[i]9    total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))10loss = total / n
    values this step0.6851790109107685 → 1.041853954849501total

  22. i ← 4

    5total = 0.06for i in range(n):7    yi = y_true[i]
    values this step3 → 4i

  23. yi ← 1

    6for i in range(n):7    yi = y_true[i]8    pi = p_pred[i]
    values this step0 → 1yi

  24. pi ← 0.8

    7yi = y_true[i]8pi = p_pred[i]9total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))
    values this step0.3 → 0.8pi

  25. total ← 1.2649975061637106

    8    pi = p_pred[i]9    total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))10loss = total / n
    values this step1.041853954849501 → 1.2649975061637106total

  26. for i in range(n):

    5total = 0.06for i in range(n):7    yi = y_true[i]

  27. loss ← 0.2529995012327421

    9    total = total - (yi * math.log(pi) + (1 - yi) * math.log(1 - pi))10loss = total / n11print('RESULT:', round(loss, 4))
    values this step0.2529995012327421loss

  28. stdout ← RESULT: 0.253

    10loss = total / n11print('RESULT:', round(loss, 4))
    values this stepRESULT: 0.253stdout

With scikit-learn

log_loss(y_true, p_pred, labels=[0,1]) uses the same natural-log formula. labels=[0,1] is passed explicitly to fix class ordering for the binary case.

library.py 
from sklearn.metrics import log_loss
from dalib.display import set_display
set_display()

y_true = [1, 0, 1, 0, 1]
p_pred = [0.9, 0.2, 0.7, 0.3, 0.8]
loss = float(log_loss(y_true, p_pred, labels=[0, 1]))
print('y_true:', y_true)
print('p_pred:', p_pred)
print('RESULT:', round(loss, 4))

y_true: [1, 0, 1, 0, 1]
p_pred: [0.9, 0.2, 0.7, 0.3, 0.8]
RESULT: 0.253

Implementation notes

  • Log loss penalises confident wrong predictions heavily: predicting p=0.01 for a true positive adds −ln(0.01)≈4.6 to the sum vs −ln(0.9)≈0.1.
  • Natural log (base e), not log₂ or log₁₀. sklearn uses the same convention.
  • sklearn clips probabilities by default (eps≈1e-15) to avoid log(0). Choosing p in (0,1) open — here [0.2,0.9] — means no clipping occurs and naive math.log matches sklearn exactly.
  • For y=1 the term reduces to −ln(p); for y=0 it reduces to −ln(1−p). The formula handles both in one expression via the y/1−y multipliers.
  • Lower log loss is better. A model predicting 0.5 everywhere gives log loss = ln(2) ≈ 0.693 regardless of labels — the random baseline.
  • Cross-reference: sigmoid-function (this chapter) produces the probabilities that log loss evaluates.