Validation Pipelines

Validate Value Ranges

Report values that fall outside an allowed [lo, hi] band. Rather than flagging every element (True/False), collect the offending values for an error report. By hand, loop with an index and append any out-of-band value. With pandas, filter with a boolean mask to get the violating rows directly.

By hand

Loop with range(len(values)) and test values[i] < lo or values[i] > hi. Collect offending values in invalid. The trace shows invalid growing only on the out-of-band iterations (index 1: −3, index 4: 89, index 6: −1).

naive.py
Replay: real traced execution (multi-file project)
values = [12, -3, 45, 7, 89, 5, -1]
lo = 0
hi = 50
invalid = []
for i in range(len(values)):
    if values[i] < lo or values[i] > hi:
        invalid.append(values[i])
print('RESULT:', invalid)

  1. values ← [12, -3, 45, 7, 89, 5, -1]

    1values = [12, -3, 45, 7, 89, 5, -1]2lo = 0
    values this step[12, -3, 45, 7, 89, 5, -1]values

  2. lo ← 0

    1values = [12, -3, 45, 7, 89, 5, -1]2lo = 03hi = 50
    values this step0lo

  3. hi ← 50

    2lo = 03hi = 504invalid = []
    values this step50hi

  4. invalid ← []

    3hi = 504invalid = []5for i in range(len(values)):
    values this step[]invalid

  5. i ← 0

    4invalid = []5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:
    values this step0i

  6. if values[i] < lo or values[i] > hi:

    5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])

  7. i ← 1

    4invalid = []5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:
    values this step0 → 1i

  8. if values[i] < lo or values[i] > hi:

    5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])

  9. invalid ← [-3]

    6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])8print('RESULT:', invalid)
    values this step[] → [-3]invalid

  10. i ← 2

    4invalid = []5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:
    values this step1 → 2i

  11. if values[i] < lo or values[i] > hi:

    5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])

  12. i ← 3

    4invalid = []5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:
    values this step2 → 3i

  13. if values[i] < lo or values[i] > hi:

    5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])

  14. i ← 4

    4invalid = []5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:
    values this step3 → 4i

  15. if values[i] < lo or values[i] > hi:

    5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])

  16. invalid ← [-3, 89]

    6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])8print('RESULT:', invalid)
    values this step[-3] → [-3, 89]invalid

  17. i ← 5

    4invalid = []5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:
    values this step4 → 5i

  18. if values[i] < lo or values[i] > hi:

    5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])

  19. i ← 6

    4invalid = []5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:
    values this step5 → 6i

  20. if values[i] < lo or values[i] > hi:

    5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])

  21. invalid ← [-3, 89, -1]

    6    if values[i] < lo or values[i] > hi:7        invalid.append(values[i])8print('RESULT:', invalid)
    values this step[-3, 89] → [-3, 89, -1]invalid

  22. for i in range(len(values)):

    4invalid = []5for i in range(len(values)):6    if values[i] < lo or values[i] > hi:

  23. stdout ← RESULT: [-3, 89, -1]

    7        invalid.append(values[i])8print('RESULT:', invalid)
    values this stepRESULT: [-3, 89, -1]stdout

With pandas

Build a boolean mask with (df['x'] < lo) | (df['x'] > hi), then use it to filter the DataFrame. The violating rows keep their original index labels, so the snapshot shows both which positions (index: [1, 4, 6]) and which values are invalid.

library.py 
import pandas as pd
from dalib.display import set_display
set_display()

values = [12, -3, 45, 7, 89, 5, -1]
lo = 0
hi = 50
df = pd.DataFrame({'x': values})
mask = (df['x'] < lo) | (df['x'] > hi)
invalid = df[mask]
result = invalid['x'].tolist()
print('lo:', lo, 'hi:', hi)
print('violations:', len(result))
print('index:', invalid.index.tolist())
print('RESULT:', result)

lo: 0 hi: 50
violations: 3
index: [1, 4, 6]
RESULT: [-3, 89, -1]

Implementation notes

  • This lesson reports offending values; range-flag (ch06) flags every element True/False. Use range-flag when you need a boolean mask for further processing; use validate-value-ranges when you need a list of errors to surface to a caller.
  • The violating rows preserve their original index, which is useful for tracing back to the source row in a larger DataFrame.
  • Cross-reference: range-flag (chapter 06) for the boolean-flag version of the same out-of-band check.