Validation Pipelines

Validate Required Columns

Check that all required columns are present in a record or DataFrame. Loop through the required names and collect any that are missing; return a (is_valid, missing) pair. By hand, use col not in record for each required name. With pandas, a set difference set(required) - set(df.columns) finds all missing names in one expression.

By hand

With pandas

set(required) - set(df.columns) is the set difference: names in required that do not appear in df.columns. Sorting the result makes the output deterministic regardless of set iteration order.

naive.py 
record = {'name': 'Alice', 'age': 30, 'score': 85}
required = ['name', 'age', 'score', 'email']
missing = []
for col in required:
    if col not in record:
        missing.append(col)
is_valid = len(missing) == 0
print('RESULT:', (is_valid, sorted(missing)))
library.py 
import pandas as pd
from dalib.display import set_display
set_display()

records = [
    {'name': 'Alice', 'age': 30, 'score': 85},
    {'name': 'Bob',   'age': 25, 'score': 92},
]
df = pd.DataFrame(records)
required = ['name', 'age', 'score', 'email']
missing = sorted(set(required) - set(df.columns))
is_valid = len(missing) == 0
result = (is_valid, missing)
print('columns:', df.columns.tolist())
print('required:', required)
print('missing:', missing)
print('RESULT:', result)

columns: ['name', 'age', 'score']
required: ['name', 'age', 'score', 'email']
missing: ['email']
RESULT: (False, ['email'])

Implementation notes

  • Sort the missing list (sorted(...)) so the output is stable — set subtraction order is not guaranteed.
  • An empty missing list means is_valid = True; the caller can then branch on the bool or raise on the list being non-empty.
  • For a valid DataFrame, set(required) - set(df.columns) returns an empty set and RESULT is (True, []).