Reshaping by Hand

Flatten Nested

A dict-of-lists in wide/nested format unrolled into flat (category, item) row tuples with a nested loop. The trace shows k and vs loading each outer key in turn, then v stepping through its values — one (k, v) pair per inner iteration.

By hand

Walk each key k and its list vs with an outer loop, then walk vs with an inner loop, appending (k, v) to rows each time. (rows is excluded from the trace because it overflows at 5 items; the outer key and current item are visible instead.)

naive.py
Replay: real traced execution (multi-file project)
data = {'fruit': ['apple', 'pear'], 'veg': ['carrot', 'pea'], 'grain': ['oat', 'rye']}
rows = []
# trace: ignore rows
for k, vs in data.items():
    for v in vs:
        rows.append((k, v))
print('RESULT:', (len(rows), rows[-1]))

  1. data ← {'fruit': ['apple', 'pear'], 'veg': ['carrot', 'pea'], 'grain': ['oat', 'rye']}

    1data = {'fruit': ['apple', 'pear'], 'veg': ['carrot', 'pea'], 'grain': ['oat', 'rye']}2rows = []
    values this step{'fruit': ['apple', 'pear'], 'veg': ['carrot', 'pea'], 'grain': ['oat', 'rye']}data

  2. rows = []

    1data = {'fruit': ['apple', 'pear'], 'veg': ['carrot', 'pea'], 'grain': ['oat', 'rye']}2rows = []3# trace: ignore rows

  3. k ← 'fruit', vs ← ['apple', 'pear']

    3# trace: ignore rows4for k, vs in data.items():5    for v in vs:
    values this step'fruit'k['apple', 'pear']vs

  4. v ← 'apple'

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))
    values this step'apple'v

  5. rows.append((k, v))

    5    for v in vs:6        rows.append((k, v))7print('RESULT:', (len(rows), rows[-1]))

  6. v ← 'pear'

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))
    values this step'apple' → 'pear'v

  7. rows.append((k, v))

    5    for v in vs:6        rows.append((k, v))7print('RESULT:', (len(rows), rows[-1]))

  8. for v in vs:

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))

  9. k ← 'veg', vs ← ['carrot', 'pea']

    3# trace: ignore rows4for k, vs in data.items():5    for v in vs:
    values this step'fruit' → 'veg'k['apple', 'pear'] → ['carrot', 'pea']vs

  10. v ← 'carrot'

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))
    values this step'pear' → 'carrot'v

  11. rows.append((k, v))

    5    for v in vs:6        rows.append((k, v))7print('RESULT:', (len(rows), rows[-1]))

  12. v ← 'pea'

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))
    values this step'carrot' → 'pea'v

  13. rows.append((k, v))

    5    for v in vs:6        rows.append((k, v))7print('RESULT:', (len(rows), rows[-1]))

  14. for v in vs:

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))

  15. k ← 'grain', vs ← ['oat', 'rye']

    3# trace: ignore rows4for k, vs in data.items():5    for v in vs:
    values this step'veg' → 'grain'k['carrot', 'pea'] → ['oat', 'rye']vs

  16. v ← 'oat'

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))
    values this step'pea' → 'oat'v

  17. rows.append((k, v))

    5    for v in vs:6        rows.append((k, v))7print('RESULT:', (len(rows), rows[-1]))

  18. v ← 'rye'

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))
    values this step'oat' → 'rye'v

  19. rows.append((k, v))

    5    for v in vs:6        rows.append((k, v))7print('RESULT:', (len(rows), rows[-1]))

  20. for v in vs:

    4for k, vs in data.items():5    for v in vs:6        rows.append((k, v))

  21. for k, vs in data.items():

    3# trace: ignore rows4for k, vs in data.items():5    for v in vs:

  22. stdout ← RESULT: (6, ('grain', 'rye'))

    6        rows.append((k, v))7print('RESULT:', (len(rows), rows[-1]))
    values this stepRESULT: (6, ('grain', 'rye'))stdout

The Pythonic way

A nested list comprehension mirrors the two loops in a single expression. The outer clause for k, vs in data.items() iterates keys; the inner clause for v in vs iterates each value list.

library.py 
data = {'fruit': ['apple', 'pear'], 'veg': ['carrot', 'pea'], 'grain': ['oat', 'rye']}
rows = [(k, v) for k, vs in data.items() for v in vs]
print('RESULT:', (len(rows), rows[-1]))

RESULT: (6, ('grain', 'rye'))

Implementation notes

  • This is the inverse of pivot-long-to-wide: where that lesson folded long (key, attr, val) rows into a nested dict-of-dicts, this lesson unfolds a nested dict-of-lists back into long (category, item) rows.
  • dict.items() returns key-value pairs in insertion order (guaranteed since Python 3.7). If the input order is not meaningful, sort the output with sorted(rows) for a deterministic result.
  • In pandas, df.melt(id_vars=[...], value_vars=[...]) performs the wide-to- long transformation. The pandas melt lesson (roadmap) covers aggregation and column naming in the melted result.