Data Pipeline Patterns
Filter Records
Filtering keeps only the records that match the rule for the next step.
Filter Records
filter_records.py
records = [
{"name": "Ada", "score": 92},
{"name": "Linus", "score": 78},
{"name": "Grace", "score": 86},
]
min_score =
passed = []
for record in records:
if record["score"] >= min_score:
passed.append(record["name"])
print("passed=" + ",".join(passed))
records = [
{"name": "Ada", "score": 92},
{"name": "Linus", "score": 78},
{"name": "Grace", "score": 86},
]
min_score =
passed = []
for record in records:
if record["score"] >= min_score:
passed.append(record["name"])
print("passed=" + ",".join(passed))
filtering records
A filter stage copies matching records into a new list while preserving their order.