Outliers and Ranges

Z-Score Flag

Flag values that are more than k standard deviations from the mean. Compute the mean and sample standard deviation of the series, calculate each value's z-score as (v − mean) / std, then mark any value with |z| > k as an outlier. By hand, use two loops (one for mean, one for std) then a third to flag. With pandas, the same operations vectorize in one expression.

By hand

First loop: accumulate total to compute mean = total / n. Second loop: accumulate squared deviations, then std = (sq_diff / (n-1)) ** 0.5. Dividing by n-1 (not n) gives the sample standard deviation — the same formula pandas .std() uses by default. Third loop: flag values where abs(v - mean) / std > k.

naive.py
Replay: real traced execution (multi-file project)
values = [5, 5, 6, 5, 5, 40]
n = len(values)
total = 0.0
for v in values:
    total = total + v
mean = total / n
sq_diff = 0.0
for v in values:
    sq_diff = sq_diff + (v - mean) ** 2
std = (sq_diff / (n - 1)) ** 0.5
k = 2
result = []
for v in values:
    result.append(abs(v - mean) / std > k)
print('RESULT:', result)

  1. values ← [5, 5, 6, 5, 5, 40]

    1values = [5, 5, 6, 5, 5, 40]2n = len(values)
    values this step[5, 5, 6, 5, 5, 40]values

  2. n ← 6

    1values = [5, 5, 6, 5, 5, 40]2n = len(values)3total = 0.0
    values this step6n

  3. total ← 0.0

    2n = len(values)3total = 0.04for v in values:
    values this step0.0total

  4. v ← 5, total ← 5.0

    pass 1 of 6
    3total = 0.04for v in values:5    total = total + v6mean = total / n
    values this step5v0.0 → 5.0total
    All 6 passes — pass 1 is the card above
    passvtotal
    150.0 → 5.0
    2—5.0 → 10.0
    35 → 610.0 → 16.0
    46 → 516.0 → 21.0
    5—21.0 → 26.0
    65 → 4026.0 → 66.0

  5. for v in values:

    3total = 0.04for v in values:5    total = total + v

  6. mean ← 11.0

    5    total = total + v6mean = total / n7sq_diff = 0.0
    values this step11.0mean

  7. sq_diff ← 0.0

    6mean = total / n7sq_diff = 0.08for v in values:
    values this step0.0sq_diff

  8. v ← 5, sq_diff ← 36.0

    pass 1 of 6
    7sq_diff = 0.08for v in values:9    sq_diff = sq_diff + (v - mean) ** 210std = (sq_diff / (n - 1)) ** 0.5
    values this step40 → 5v0.0 → 36.0sq_diff
    All 6 passes — pass 1 is the card above
    passvsq_diff
    140 → 50.0 → 36.0
    2—36.0 → 72.0
    35 → 672.0 → 97.0
    46 → 597.0 → 133.0
    5—133.0 → 169.0
    65 → 40169.0 → 1010.0

  9. for v in values:

    7sq_diff = 0.08for v in values:9    sq_diff = sq_diff + (v - mean) ** 2

  10. std ← 14.212670403551895

    9    sq_diff = sq_diff + (v - mean) ** 210std = (sq_diff / (n - 1)) ** 0.511k = 2
    values this step14.212670403551895std

  11. k ← 2

    10std = (sq_diff / (n - 1)) ** 0.511k = 212result = []
    values this step2k

  12. result ← []

    11k = 212result = []13for v in values:
    values this step[]result

  13. v ← 5, result ← [False]

    pass 1 of 6
    12result = []13for v in values:14    result.append(abs(v - mean) / std > k)15print('RESULT:', result)
    values this step40 → 5v[] → [False]result
    All 6 passes — pass 1 is the card above
    passvresult
    140 → 5[] → [False]
    2—[False] → [False, False]
    35 → 6[False, False] → [False, False, False]
    46 → 5[False, False, False] → [False, False, False, False]
    5—[False, False, False, False] → [False, False, False, False, False]
    65 → 40[False, False, False, False, False] → [False, False, False, False, False, True]

  14. for v in values:

    12result = []13for v in values:14    result.append(abs(v - mean) / std > k)

  15. stdout ← RESULT: [False, False, False, False, False, True]

    14    result.append(abs(v - mean) / std > k)15print('RESULT:', result)
    values this stepRESULT: [False, False, False, False, False, True]stdout

With pandas

df['x'].mean() and df['x'].std() compute the mean and sample standard deviation (ddof=1). Subtracting the mean and dividing by std gives a Series of z-scores; .abs() > k produces the boolean flags.

library.py 
import pandas as pd
from dalib.display import set_display
set_display()

values = [5, 5, 6, 5, 5, 40]
k = 2
df = pd.DataFrame({'x': values})
mean = df['x'].mean()
std = df['x'].std()
z = (df['x'] - mean) / std
flags = z.abs() > k
result = flags.tolist()
print('mean:', round(mean, 4))
print('std:', round(std, 4))
print('z:', [round(v, 2) for v in z.tolist()])
print('RESULT:', result)

mean: 11.0
std: 14.2127
z: [-0.42, -0.42, -0.35, -0.42, -0.42, 2.04]
RESULT: [False, False, False, False, False, True]

Implementation notes

  • pandas .std() uses ddof=1 (sample std, divides by n−1) by default. numpy .std() uses ddof=0 (population std, divides by n). Using the wrong ddof produces a slightly different std and breaks parity — the naive half must match whichever the library call uses.
  • k=2 is a common threshold; k=3 is stricter and flags fewer values.
  • Z-scores assume the data is roughly normally distributed; IQR fences (iqr-outlier-flags, this chapter) are more robust to heavy-tailed distributions.
  • Cross-reference: stats-z-scores (statistics chapter) for standardization and the normal distribution context.