Validation Pipelines

Validate Required Columns

Check that all required columns are present in a record or DataFrame. Loop through the required names and collect any that are missing; return a (is_valid, missing) pair. By hand, use col not in record for each required name. With pandas, a set difference set(required) - set(df.columns) finds all missing names in one expression.

By hand

Walk required and append any name not found in record to missing. is_valid is True only when missing is empty. The trace shows missing growing if any column is absent — here 'email' is not in record, so it is the sole missing entry.

naive.py
Replay: real traced execution (multi-file project)
record = {'name': 'Alice', 'age': 30, 'score': 85}
required = ['name', 'age', 'score', 'email']
missing = []
for col in required:
    if col not in record:
        missing.append(col)
is_valid = len(missing) == 0
print('RESULT:', (is_valid, sorted(missing)))

  1. record ← {'name': 'Alice', 'age': 30, 'score': 85}

    1record = {'name': 'Alice', 'age': 30, 'score': 85}2required = ['name', 'age', 'score', 'email']
    values this step{'name': 'Alice', 'age': 30, 'score': 85}record

  2. required ← ['name', 'age', 'score', 'email']

    1record = {'name': 'Alice', 'age': 30, 'score': 85}2required = ['name', 'age', 'score', 'email']3missing = []
    values this step['name', 'age', 'score', 'email']required

  3. missing ← []

    2required = ['name', 'age', 'score', 'email']3missing = []4for col in required:
    values this step[]missing

  4. col ← 'name'

    3missing = []4for col in required:5    if col not in record:
    values this step'name'col

  5. if col not in record:

    4for col in required:5    if col not in record:6        missing.append(col)

  6. col ← 'age'

    3missing = []4for col in required:5    if col not in record:
    values this step'name' → 'age'col

  7. if col not in record:

    4for col in required:5    if col not in record:6        missing.append(col)

  8. col ← 'score'

    3missing = []4for col in required:5    if col not in record:
    values this step'age' → 'score'col

  9. if col not in record:

    4for col in required:5    if col not in record:6        missing.append(col)

  10. col ← 'email'

    3missing = []4for col in required:5    if col not in record:
    values this step'score' → 'email'col

  11. if col not in record:

    4for col in required:5    if col not in record:6        missing.append(col)

  12. missing ← ['email']

    5    if col not in record:6        missing.append(col)7is_valid = len(missing) == 0
    values this step[] → ['email']missing

  13. for col in required:

    3missing = []4for col in required:5    if col not in record:

  14. is_valid ← False

    6        missing.append(col)7is_valid = len(missing) == 08print('RESULT:', (is_valid, sorted(missing)))
    values this stepFalseis_valid

  15. stdout ← RESULT: (False, ['email'])

    7is_valid = len(missing) == 08print('RESULT:', (is_valid, sorted(missing)))
    values this stepRESULT: (False, ['email'])stdout

With pandas

set(required) - set(df.columns) is the set difference: names in required that do not appear in df.columns. Sorting the result makes the output deterministic regardless of set iteration order.

library.py 
import pandas as pd
from dalib.display import set_display
set_display()

records = [
    {'name': 'Alice', 'age': 30, 'score': 85},
    {'name': 'Bob',   'age': 25, 'score': 92},
]
df = pd.DataFrame(records)
required = ['name', 'age', 'score', 'email']
missing = sorted(set(required) - set(df.columns))
is_valid = len(missing) == 0
result = (is_valid, missing)
print('columns:', df.columns.tolist())
print('required:', required)
print('missing:', missing)
print('RESULT:', result)

columns: ['name', 'age', 'score']
required: ['name', 'age', 'score', 'email']
missing: ['email']
RESULT: (False, ['email'])

Implementation notes

  • Sort the missing list (sorted(...)) so the output is stable — set subtraction order is not guaranteed.
  • An empty missing list means is_valid = True; the caller can then branch on the bool or raise on the list being non-empty.
  • For a valid DataFrame, set(required) - set(df.columns) returns an empty set and RESULT is (True, []).