Reshaping

Melt Wide to Long

Unpivot wide records (id + attribute columns) into long (id, variable, value) triples — one row per attribute per original record. With pandas, pd.melt does this in one call. Melt is the inverse of pivot.

By hand

Loop over each wide record, then over each attribute name. Each combination emits one (id, attr, value) tuple into long. The result list grows one entry per attribute per record.

naive.py
Replay: real traced execution (multi-file project)
# trace: ignore records
records = [
    {'id': 'p', 'a': 10, 'b': 20},
    {'id': 'q', 'a': 30, 'b': 40},
]
attrs = ['a', 'b']
long = []
for r in records:
    for attr in attrs:
        long.append((r['id'], attr, r[attr]))
print('RESULT:', long)

  1. {'id': 'p', 'a': 10, 'b': 20},

    2records = [3    {'id': 'p', 'a': 10, 'b': 20},4    {'id': 'q', 'a': 30, 'b': 40},

  2. {'id': 'q', 'a': 30, 'b': 40},

    3    {'id': 'p', 'a': 10, 'b': 20},4    {'id': 'q', 'a': 30, 'b': 40},5]

  3. records = [

    1# trace: ignore records2records = [3    {'id': 'p', 'a': 10, 'b': 20},

  4. attrs ← ['a', 'b']

    5]6attrs = ['a', 'b']7long = []
    values this step['a', 'b']attrs

  5. long ← []

    6attrs = ['a', 'b']7long = []8for r in records:
    values this step[]long

  6. r ← {'id': 'p', 'a': 10, 'b': 20}

    7long = []8for r in records:9    for attr in attrs:
    values this step{'id': 'p', 'a': 10, 'b': 20}r

  7. attr ← 'a'

    8for r in records:9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))
    values this step'a'attr

  8. long ← [('p', 'a', 10)]

    9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))11print('RESULT:', long)
    values this step[] → [('p', 'a', 10)]long

  9. attr ← 'b'

    8for r in records:9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))
    values this step'a' → 'b'attr

  10. long ← [('p', 'a', 10), ('p', 'b', 20)]

    9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))11print('RESULT:', long)
    values this step[('p', 'a', 10)] → [('p', 'a', 10), ('p', 'b', 20)]long

  11. for attr in attrs:

    8for r in records:9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))

  12. r ← {'id': 'q', 'a': 30, 'b': 40}

    7long = []8for r in records:9    for attr in attrs:
    values this step{'id': 'p', 'a': 10, 'b': 20} → {'id': 'q', 'a': 30, 'b': 40}r

  13. attr ← 'a'

    8for r in records:9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))
    values this step'b' → 'a'attr

  14. long ← [('p', 'a', 10), ('p', 'b', 20), ('q', 'a', 30)]

    9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))11print('RESULT:', long)
    values this step[('p', 'a', 10), ('p', 'b', 20)] → [('p', 'a', 10), ('p', 'b', 20), ('q', 'a', 30)]long

  15. attr ← 'b'

    8for r in records:9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))
    values this step'a' → 'b'attr

  16. long ← [('p', 'a', 10), ('p', 'b', 20), ('q', 'a', 30), ('q', 'b', 40)]

    9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))11print('RESULT:', long)
    values this step[('p', 'a', 10), ('p', 'b', 20), ('q', 'a', 30)] → [('p', 'a', 10), ('p', 'b', 20), ('q', 'a', 30), ('q', 'b', 40)]long

  17. for attr in attrs:

    8for r in records:9    for attr in attrs:10        long.append((r['id'], attr, r[attr]))

  18. for r in records:

    7long = []8for r in records:9    for attr in attrs:

  19. stdout ← RESULT: [('p', 'a', 10), ('p', 'b', 20), ('q', 'a', 30), ('q', 'b', 40)]

    10        long.append((r['id'], attr, r[attr]))11print('RESULT:', long)
    values this stepRESULT: [('p', 'a', 10), ('p', 'b', 20), ('q', 'a', 30), ('q', 'b', 40)]stdout

With pandas

pd.melt(df, id_vars='id', value_vars=['a', 'b']) unpivots both attribute columns into variable and value rows. Pandas emits all rows for 'a' before all rows for 'b' (column-major order); the naive loop is row-major. Sorting makes both sides comparable.

library.py 
import pandas as pd
from dalib.display import set_display
set_display()

records = [
    {'id': 'p', 'a': 10, 'b': 20},
    {'id': 'q', 'a': 30, 'b': 40},
]
df = pd.DataFrame(records)
melted = pd.melt(df, id_vars='id', value_vars=['a', 'b'])
result = list(zip(
    melted['id'].tolist(),
    melted['variable'].tolist(),
    melted['value'].tolist(),
))
print('columns:', melted.columns.tolist())
print('id:', melted['id'].tolist())
print('variable:', melted['variable'].tolist())
print('value:', melted['value'].tolist())
print('RESULT:', sorted(result))

columns: ['id', 'variable', 'value']
id: ['p', 'q', 'p', 'q']
variable: ['a', 'a', 'b', 'b']
value: [10, 30, 20, 40]
RESULT: [('p', 'a', 10), ('p', 'b', 20), ('q', 'a', 30), ('q', 'b', 40)]

Implementation notes

  • pd.melt stacks value_vars in column-major order: all rows for 'a' come before all rows for 'b'. The naive loop is row-major (all attributes of 'p' before all attributes of 'q'). Both represent the same data; sort to compare.
  • Omit value_vars to melt every column not in id_vars.
  • The generated column names default to 'variable' and 'value'; rename them with var_name='attr' and value_name='score'.
  • Cross-reference: pivot-table-simple (this chapter) for the inverse long-to-wide direction; flatten-nested (python-data-basics) for pure-Python unpivoting.