Categorical Encoding

Collapse Rare Categories

Reduce cardinality by replacing low-frequency categories with a single 'other' label. Count how often each category appears, then remap any category with a count below the threshold. By hand, use a frequency dict and a conditional in a loop. With pandas, use value_counts() to identify frequent categories and where() to replace the rest.

By hand

First loop: build counts via counts.get(s, 0) + 1. The trace shows the dict filling in with running frequencies. Second loop: append the original label when counts[s] >= threshold, otherwise append 'other'. With threshold=2, only cat (4×) and dog (2×) survive; bird (1×) and fish (1×) become 'other'.

naive.py
Replay: real traced execution (multi-file project)
labels = ['cat', 'dog', 'cat', 'bird', 'cat', 'dog', 'fish', 'cat']
threshold = 2
counts = {}
for s in labels:
    counts[s] = counts.get(s, 0) + 1
result = []
for s in labels:
    result.append(s if counts[s] >= threshold else 'other')
print('RESULT:', result)

  1. labels ← ['cat', 'dog', 'cat', 'bird', 'cat', 'dog', 'fish', 'cat']

    1labels = ['cat', 'dog', 'cat', 'bird', 'cat', 'dog', 'fish', 'cat']2threshold = 2
    values this step['cat', 'dog', 'cat', 'bird', 'cat', 'dog', 'fish', 'cat']labels

  2. threshold ← 2

    1labels = ['cat', 'dog', 'cat', 'bird', 'cat', 'dog', 'fish', 'cat']2threshold = 23counts = {}
    values this step2threshold

  3. counts ← {}

    2threshold = 23counts = {}4for s in labels:
    values this step{}counts

  4. s ← 'cat', counts ← {'cat': 1}

    pass 1 of 8
    3counts = {}4for s in labels:5    counts[s] = counts.get(s, 0) + 16result = []
    values this step'cat's{} → {'cat': 1}counts
    All 8 passes — pass 1 is the card above
    passscounts
    1'cat'{} → {'cat': 1}
    2'cat' → 'dog'{'cat': 1} → {'cat': 1, 'dog': 1}
    3'dog' → 'cat'{'cat': 1, 'dog': 1} → {'cat': 2, 'dog': 1}
    4'cat' → 'bird'{'cat': 2, 'dog': 1} → {'cat': 2, 'dog': 1, 'bird': 1}
    5'bird' → 'cat'{'cat': 2, 'dog': 1, 'bird': 1} → {'cat': 3, 'dog': 1, 'bird': 1}
    6'cat' → 'dog'{'cat': 3, 'dog': 1, 'bird': 1} → {'cat': 3, 'dog': 2, 'bird': 1}
    7'dog' → 'fish'{'cat': 3, 'dog': 2, 'bird': 1} → {'cat': 3, 'dog': 2, 'bird': 1, 'fish': 1}
    8'fish' → 'cat'{'cat': 3, 'dog': 2, 'bird': 1, 'fish': 1} → {'cat': 4, 'dog': 2, 'bird': 1, 'fish': 1}

  5. for s in labels:

    3counts = {}4for s in labels:5    counts[s] = counts.get(s, 0) + 1

  6. result ← []

    5    counts[s] = counts.get(s, 0) + 16result = []7for s in labels:
    values this step[]result

  7. result ← ['cat']

    pass 1 of 8
    6result = []7for s in labels:8    result.append(s if counts[s] >= threshold else 'other')9print('RESULT:', result)
    values this step[] → ['cat']result
    All 8 passes — pass 1 is the card above
    passresults
    1[] → ['cat']—
    2['cat'] → ['cat', 'dog']'cat' → 'dog'
    3['cat', 'dog'] → ['cat', 'dog', 'cat']'dog' → 'cat'
    4['cat', 'dog', 'cat'] → ['cat', 'dog', 'cat', 'other']'cat' → 'bird'
    5['cat', 'dog', 'cat', 'other'] → ['cat', 'dog', 'cat', 'other', 'cat']'bird' → 'cat'
    6['cat', 'dog', 'cat', 'other', 'cat'] → ['cat', 'dog', 'cat', 'other', 'cat', 'dog']'cat' → 'dog'
    7['cat', 'dog', 'cat', 'other', 'cat', 'dog'] → ['cat', 'dog', 'cat', 'other', 'cat', 'dog', 'other']'dog' → 'fish'
    8['cat', 'dog', 'cat', 'other', 'cat', 'dog', 'other'] → ['cat', 'dog', 'cat', 'other', 'cat', 'dog', 'other', 'cat']'fish' → 'cat'

  8. for s in labels:

    6result = []7for s in labels:8    result.append(s if counts[s] >= threshold else 'other')

  9. stdout ← RESULT: ['cat', 'dog', 'cat', 'other', 'cat', 'dog', 'other', 'cat']

    8    result.append(s if counts[s] >= threshold else 'other')9print('RESULT:', result)
    values this stepRESULT: ['cat', 'dog', 'cat', 'other', 'cat', 'dog', 'other', 'cat']stdout

With pandas

value_counts() returns frequencies in descending order. Index into it with freq >= threshold to get the frequent category labels, then use df['c'].where(df['c'].isin(frequent), 'other') to keep frequent values and replace the rest.

library.py 
import pandas as pd
from dalib.display import set_display
set_display()

labels = ['cat', 'dog', 'cat', 'bird', 'cat', 'dog', 'fish', 'cat']
threshold = 2
df = pd.DataFrame({'c': labels})
freq = df['c'].value_counts()
frequent = freq[freq >= threshold].index
s = df['c'].where(df['c'].isin(frequent), 'other')
result = s.tolist()
print('counts:', freq.to_dict())
print('frequent:', sorted(frequent.tolist()))
print('RESULT:', result)

counts: {'cat': 4, 'dog': 2, 'bird': 1, 'fish': 1}
frequent: ['cat', 'dog']
RESULT: ['cat', 'dog', 'cat', 'other', 'cat', 'dog', 'other', 'cat']

Implementation notes

  • The threshold is a design choice: >= 2 here keeps anything that appears more than once; stricter thresholds produce fewer surviving categories.
  • where(condition, other) keeps elements where condition is True and replaces them with other where it is False — the opposite of the intuitive reading of "where fish is rare, use other".
  • Cross-reference: frequency-count (python-data-basics) for the general frequency-counting pattern this lesson builds on.