Validation Pipelines

Validate Value Ranges

Report values that fall outside an allowed [lo, hi] band. Rather than flagging every element (True/False), collect the offending values for an error report. By hand, loop with an index and append any out-of-band value. With pandas, filter with a boolean mask to get the violating rows directly.

By hand

With pandas

Build a boolean mask with (df['x'] < lo) | (df['x'] > hi), then use it to filter the DataFrame. The violating rows keep their original index labels, so the snapshot shows both which positions (index: [1, 4, 6]) and which values are invalid.

naive.py 
values = [12, -3, 45, 7, 89, 5, -1]
lo = 0
hi = 50
invalid = []
for i in range(len(values)):
    if values[i] < lo or values[i] > hi:
        invalid.append(values[i])
print('RESULT:', invalid)
library.py 
import pandas as pd
from dalib.display import set_display
set_display()

values = [12, -3, 45, 7, 89, 5, -1]
lo = 0
hi = 50
df = pd.DataFrame({'x': values})
mask = (df['x'] < lo) | (df['x'] > hi)
invalid = df[mask]
result = invalid['x'].tolist()
print('lo:', lo, 'hi:', hi)
print('violations:', len(result))
print('index:', invalid.index.tolist())
print('RESULT:', result)

lo: 0 hi: 50
violations: 3
index: [1, 4, 6]
RESULT: [-3, 89, -1]

Implementation notes

  • This lesson reports offending values; range-flag (ch06) flags every element True/False. Use range-flag when you need a boolean mask for further processing; use validate-value-ranges when you need a list of errors to surface to a caller.
  • The violating rows preserve their original index, which is useful for tracing back to the source row in a larger DataFrame.
  • Cross-reference: range-flag (chapter 06) for the boolean-flag version of the same out-of-band check.