Outliers and Ranges

Z-Score Flag

Flag values that are more than k standard deviations from the mean. Compute the mean and sample standard deviation of the series, calculate each value's z-score as (v āˆ’ mean) / std, then mark any value with |z| > k as an outlier. By hand, use two loops (one for mean, one for std) then a third to flag. With pandas, the same operations vectorize in one expression.

By hand

With pandas

df['x'].mean() and df['x'].std() compute the mean and sample standard deviation (ddof=1). Subtracting the mean and dividing by std gives a Series of z-scores; .abs() > k produces the boolean flags.

naive.py 
values = [5, 5, 6, 5, 5, 40]
n = len(values)
total = 0.0
for v in values:
    total = total + v
mean = total / n
sq_diff = 0.0
for v in values:
    sq_diff = sq_diff + (v - mean) ** 2
std = (sq_diff / (n - 1)) ** 0.5
k = 2
result = []
for v in values:
    result.append(abs(v - mean) / std > k)
print('RESULT:', result)
library.py 
import pandas as pd
from dalib.display import set_display
set_display()

values = [5, 5, 6, 5, 5, 40]
k = 2
df = pd.DataFrame({'x': values})
mean = df['x'].mean()
std = df['x'].std()
z = (df['x'] - mean) / std
flags = z.abs() > k
result = flags.tolist()
print('mean:', round(mean, 4))
print('std:', round(std, 4))
print('z:', [round(v, 2) for v in z.tolist()])
print('RESULT:', result)

mean: 11.0
std: 14.2127
z: [-0.42, -0.42, -0.35, -0.42, -0.42, 2.04]
RESULT: [False, False, False, False, False, True]

Implementation notes

  • pandas .std() uses ddof=1 (sample std, divides by nāˆ’1) by default. numpy .std() uses ddof=0 (population std, divides by n). Using the wrong ddof produces a slightly different std and breaks parity ā€” the naive half must match whichever the library call uses.
  • k=2 is a common threshold; k=3 is stricter and flags fewer values.
  • Z-scores assume the data is roughly normally distributed; IQR fences (iqr-outlier-flags, this chapter) are more robust to heavy-tailed distributions.
  • Cross-reference: stats-z-scores (statistics chapter) for standardization and the normal distribution context.