Categorical Encoding

Collapse Rare Categories

Reduce cardinality by replacing low-frequency categories with a single 'other' label. Count how often each category appears, then remap any category with a count below the threshold. By hand, use a frequency dict and a conditional in a loop. With pandas, use value_counts() to identify frequent categories and where() to replace the rest.

By hand

With pandas

value_counts() returns frequencies in descending order. Index into it with freq >= threshold to get the frequent category labels, then use df['c'].where(df['c'].isin(frequent), 'other') to keep frequent values and replace the rest.

naive.py 
labels = ['cat', 'dog', 'cat', 'bird', 'cat', 'dog', 'fish', 'cat']
threshold = 2
counts = {}
for s in labels:
    counts[s] = counts.get(s, 0) + 1
result = []
for s in labels:
    result.append(s if counts[s] >= threshold else 'other')
print('RESULT:', result)
library.py 
import pandas as pd
from dalib.display import set_display
set_display()

labels = ['cat', 'dog', 'cat', 'bird', 'cat', 'dog', 'fish', 'cat']
threshold = 2
df = pd.DataFrame({'c': labels})
freq = df['c'].value_counts()
frequent = freq[freq >= threshold].index
s = df['c'].where(df['c'].isin(frequent), 'other')
result = s.tolist()
print('counts:', freq.to_dict())
print('frequent:', sorted(frequent.tolist()))
print('RESULT:', result)

counts: {'cat': 4, 'dog': 2, 'bird': 1, 'fish': 1}
frequent: ['cat', 'dog']
RESULT: ['cat', 'dog', 'cat', 'other', 'cat', 'dog', 'other', 'cat']

Implementation notes

  • The threshold is a design choice: >= 2 here keeps anything that appears more than once; stricter thresholds produce fewer surviving categories.
  • where(condition, other) keeps elements where condition is True and replaces them with other where it is False — the opposite of the intuitive reading of "where fish is rare, use other".
  • Cross-reference: frequency-count (python-data-basics) for the general frequency-counting pattern this lesson builds on.