Categorical Encoding

Map Category Spellings

Collapse spelling and case variants to a single canonical label. A variant→canonical dict maps every known form (lowercased) to the standard value. By hand, lowercase each label and look it up in the dict. With pandas, chain .str.lower() with .map(dict) to do the same in one expression.

By hand

Lowercase each label and look it up in variant_to_canonical. The dict keys are already lowercase, so the lookup always succeeds for any known variant regardless of the original casing. The trace shows result growing with canonical values as each variant is resolved.

naive.py
Replay: real traced execution (multi-file project)
variant_to_canonical = {
    'usa': 'US', 'u.s.a.': 'US', 'united states': 'US',
    'uk': 'UK',
}
labels = ['USA', 'usa', 'U.S.A.', 'United States', 'uk', 'UK']
result = []
for s in labels:
    result.append(variant_to_canonical[s.lower()])
print('RESULT:', result)

  1. 'usa': 'US', 'u.s.a.': 'US', 'united states': 'US',

    1variant_to_canonical = {2    'usa': 'US', 'u.s.a.': 'US', 'united states': 'US',3    'uk': 'UK',

  2. 'uk': 'UK',

    2    'usa': 'US', 'u.s.a.': 'US', 'united states': 'US',3    'uk': 'UK',4}

  3. variant_to_canonical ← {'usa': 'US', 'u.s.a.': 'US', 'united states': 'US', 'uk': 'UK'}

    1variant_to_canonical = {2    'usa': 'US', 'u.s.a.': 'US', 'united states': 'US',
    values this step{'usa': 'US', 'u.s.a.': 'US', 'united states': 'US', 'uk': 'UK'}variant_to_canonical

  4. labels ← ['USA', 'usa', 'U.S.A.', 'United States', 'uk', 'UK']

    4}5labels = ['USA', 'usa', 'U.S.A.', 'United States', 'uk', 'UK']6result = []
    values this step['USA', 'usa', 'U.S.A.', 'United States', 'uk', 'UK']labels

  5. result ← []

    5labels = ['USA', 'usa', 'U.S.A.', 'United States', 'uk', 'UK']6result = []7for s in labels:
    values this step[]result

  6. s ← 'USA'

    6result = []7for s in labels:8    result.append(variant_to_canonical[s.lower()])
    values this step'USA's

  7. result ← ['US']

    7for s in labels:8    result.append(variant_to_canonical[s.lower()])9print('RESULT:', result)
    values this step[] → ['US']result

  8. s ← 'usa'

    6result = []7for s in labels:8    result.append(variant_to_canonical[s.lower()])
    values this step'USA' → 'usa's

  9. result ← ['US', 'US']

    7for s in labels:8    result.append(variant_to_canonical[s.lower()])9print('RESULT:', result)
    values this step['US'] → ['US', 'US']result

  10. s ← 'U.S.A.'

    6result = []7for s in labels:8    result.append(variant_to_canonical[s.lower()])
    values this step'usa' → 'U.S.A.'s

  11. result ← ['US', 'US', 'US']

    7for s in labels:8    result.append(variant_to_canonical[s.lower()])9print('RESULT:', result)
    values this step['US', 'US'] → ['US', 'US', 'US']result

  12. s ← 'United States'

    6result = []7for s in labels:8    result.append(variant_to_canonical[s.lower()])
    values this step'U.S.A.' → 'United States's

  13. result ← ['US', 'US', 'US', 'US']

    7for s in labels:8    result.append(variant_to_canonical[s.lower()])9print('RESULT:', result)
    values this step['US', 'US', 'US'] → ['US', 'US', 'US', 'US']result

  14. s ← 'uk'

    6result = []7for s in labels:8    result.append(variant_to_canonical[s.lower()])
    values this step'United States' → 'uk's

  15. result ← ['US', 'US', 'US', 'US', 'UK']

    7for s in labels:8    result.append(variant_to_canonical[s.lower()])9print('RESULT:', result)
    values this step['US', 'US', 'US', 'US'] → ['US', 'US', 'US', 'US', 'UK']result

  16. s ← 'UK'

    6result = []7for s in labels:8    result.append(variant_to_canonical[s.lower()])
    values this step'uk' → 'UK's

  17. result ← ['US', 'US', 'US', 'US', 'UK', 'UK']

    7for s in labels:8    result.append(variant_to_canonical[s.lower()])9print('RESULT:', result)
    values this step['US', 'US', 'US', 'US', 'UK'] → ['US', 'US', 'US', 'US', 'UK', 'UK']result

  18. for s in labels:

    6result = []7for s in labels:8    result.append(variant_to_canonical[s.lower()])

  19. stdout ← RESULT: ['US', 'US', 'US', 'US', 'UK', 'UK']

    8    result.append(variant_to_canonical[s.lower()])9print('RESULT:', result)
    values this stepRESULT: ['US', 'US', 'US', 'US', 'UK', 'UK']stdout

With pandas

.str.lower() lowercases the column in one pass, then .map(dict) remaps each element to its canonical value. All inputs have a dict entry, so no NaN appears in the result.

library.py 
import pandas as pd
from dalib.display import set_display
set_display()

variant_to_canonical = {
    'usa': 'US', 'u.s.a.': 'US', 'united states': 'US',
    'uk': 'UK',
}
labels = ['USA', 'usa', 'U.S.A.', 'United States', 'uk', 'UK']
df = pd.DataFrame({'c': labels})
s = df['c'].str.lower().map(variant_to_canonical)
result = s.tolist()
print('index:', s.index.tolist())
print('dtype:', s.dtype)
print('RESULT:', result)

index: [0, 1, 2, 3, 4, 5]
dtype: object
RESULT: ['US', 'US', 'US', 'US', 'UK', 'UK']

Implementation notes

  • Lowercase before the lookup so the dict only needs one entry per variant, not one per case combination.
  • If an input has no dict entry, .map(dict) returns NaN for that element rather than raising — add a .fillna() or verify coverage before mapping.
  • Cross-reference: map-values (python-pandas ch08) for the general elementwise remap pattern this lesson applies to canonicalization.