Clean and Summarize - Data Cleaning Step by Step

Capstone: chain cleaning and aggregation. Given parallel category and value lists with one missing value, drop the row with the missing value, then produce a per-category sum. By hand, skip None values while accumulating into a running total dict. With pandas, chain dropna → groupby → sum in three calls.

By hand

A single loop walks both lists by index. When values[i] is not None, accumulate into total[cats[i]] using .get(c, 0) + values[i]. The trace shows total growing correctly — the None at index 2 is skipped entirely, so only five of the six rows contribute. dict(sorted(...)) produces a deterministic key order for the final result.

                            naive.py
                        
Replay: real traced execution (multi-file project)

                            cats   = ['A', 'B', 'A', 'B', 'A', 'B']
values = [10, 20, None, 30, 15, 25]
total = {}
for i in range(len(cats)):
    if values[i] is not None:
        c = cats[i]
        total[c] = total.get(c, 0) + values[i]
result = dict(sorted(total.items()))
print('RESULT:', result)

                        
1cats ← ['A', 'B', 'A', 'B', 'A', 'B']1cats   = ['A', 'B', 'A', 'B', 'A', 'B']2values = [10, 20, None, 30, 15, 25]⋯values this step['A', 'B', 'A', 'B', 'A', 'B']cats
2values ← [10, 20, None, 30, 15, 25]1cats   = ['A', 'B', 'A', 'B', 'A', 'B']2values = [10, 20, None, 30, 15, 25]3total = {}⋯values this step[10, 20, None, 30, 15, 25]values
3total ← {}⋯2values = [10, 20, None, 30, 15, 25]3total = {}4for i in range(len(cats)):⋯values this step{}total
4i ← 0⋯3total = {}4for i in range(len(cats)):5    if values[i] is not None:⋯values this step0i
5if values[i] is not None:⋯4for i in range(len(cats)):5    if values[i] is not None:6        c = cats[i]⋯
6c ← 'A'⋯5if values[i] is not None:6    c = cats[i]7    total[c] = total.get(c, 0) + values[i]⋯values this step'A'c
7total ← {'A': 10}⋯6        c = cats[i]7        total[c] = total.get(c, 0) + values[i]8result = dict(sorted(total.items()))⋯values this step{} → {'A': 10}total
8i ← 1⋯3total = {}4for i in range(len(cats)):5    if values[i] is not None:⋯values this step0 → 1i
9if values[i] is not None:⋯4for i in range(len(cats)):5    if values[i] is not None:6        c = cats[i]⋯
10c ← 'B'⋯5if values[i] is not None:6    c = cats[i]7    total[c] = total.get(c, 0) + values[i]⋯values this step'A' → 'B'c
11total ← {'A': 10, 'B': 20}⋯6        c = cats[i]7        total[c] = total.get(c, 0) + values[i]8result = dict(sorted(total.items()))⋯values this step{'A': 10} → {'A': 10, 'B': 20}total
12i ← 2⋯3total = {}4for i in range(len(cats)):5    if values[i] is not None:⋯values this step1 → 2i
13if values[i] is not None:⋯4for i in range(len(cats)):5    if values[i] is not None:6        c = cats[i]⋯
14i ← 3⋯3total = {}4for i in range(len(cats)):5    if values[i] is not None:⋯values this step2 → 3i
15if values[i] is not None:⋯4for i in range(len(cats)):5    if values[i] is not None:6        c = cats[i]⋯
16c = cats[i]⋯5if values[i] is not None:6    c = cats[i]7    total[c] = total.get(c, 0) + values[i]⋯
17total ← {'A': 10, 'B': 50}⋯6        c = cats[i]7        total[c] = total.get(c, 0) + values[i]8result = dict(sorted(total.items()))⋯values this step{'A': 10, 'B': 20} → {'A': 10, 'B': 50}total
18i ← 4⋯3total = {}4for i in range(len(cats)):5    if values[i] is not None:⋯values this step3 → 4i
19if values[i] is not None:⋯4for i in range(len(cats)):5    if values[i] is not None:6        c = cats[i]⋯
20c ← 'A'⋯5if values[i] is not None:6    c = cats[i]7    total[c] = total.get(c, 0) + values[i]⋯values this step'B' → 'A'c
21total ← {'A': 25, 'B': 50}⋯6        c = cats[i]7        total[c] = total.get(c, 0) + values[i]8result = dict(sorted(total.items()))⋯values this step{'A': 10, 'B': 50} → {'A': 25, 'B': 50}total
22i ← 5⋯3total = {}4for i in range(len(cats)):5    if values[i] is not None:⋯values this step4 → 5i
23if values[i] is not None:⋯4for i in range(len(cats)):5    if values[i] is not None:6        c = cats[i]⋯
24c ← 'B'⋯5if values[i] is not None:6    c = cats[i]7    total[c] = total.get(c, 0) + values[i]⋯values this step'A' → 'B'c
25total ← {'A': 25, 'B': 75}⋯6        c = cats[i]7        total[c] = total.get(c, 0) + values[i]8result = dict(sorted(total.items()))⋯values this step{'A': 25, 'B': 50} → {'A': 25, 'B': 75}total
26for i in range(len(cats)):⋯3total = {}4for i in range(len(cats)):5    if values[i] is not None:⋯
27result ← {'A': 25, 'B': 75}⋯7        total[c] = total.get(c, 0) + values[i]8result = dict(sorted(total.items()))9print('RESULT:', result)values this step{'A': 25, 'B': 75}result
28stdout ← RESULT: {'A': 25, 'B': 75}⋯8result = dict(sorted(total.items()))9print('RESULT:', result)values this stepRESULT: {'A': 25, 'B': 75}stdout

With pandas

df.dropna(subset=['val']) removes the row where val is NaN (one row dropped: shape goes from (6, 2) to (5, 2)). groupby('cat')['val'].sum() aggregates per category. The sum comes back as float64 because the column dtype is float (NaN forces float promotion); int(v) converts each value back to a plain integer to match the naive result.

                            library.py
                        

                            import pandas as pd
from dalib.display import set_display
set_display()

cats   = ['A', 'B', 'A', 'B', 'A', 'B']
values = [10, 20, None, 30, 15, 25]
df = pd.DataFrame({'cat': cats, 'val': values})
clean = df.dropna(subset=['val'])
s = clean.groupby('cat')['val'].sum()
result = {k: int(v) for k, v in sorted(s.items())}
print('shape before:', df.shape)
print('shape after:', clean.shape)
print('groups:', s.to_dict())
print('RESULT:', result)

                        

shape before: (6, 2)
shape after: (5, 2)
groups: {'A': 25.0, 'B': 75.0}
RESULT: {'A': 25, 'B': 75}

Implementation notes

A column containing None in a Python list becomes float64 in pandas (NaN forces float promotion). That is why sum() returns floats here even though all real values are integers; int() converts them back for a clean result.
Cross-reference: drop-missing-rows (ch01) for the dropna step that removes the None row before aggregation.