Merges

Counting Adjacent Pairs

The first calculation is an integer pair-count table. Frequencies weight each adjacent occurrence, so the displayed corpus determines every count and the merge rule has an exact source.

highlighted = computed this step

Counting adjacent pairs

A pair count is the sum over words of frequency times adjacent occurrences. Every count is an integer because the corpus frequencies and occurrences are integers. This is the central BPE mechanic: the current segmentation determines which adjacent pairs exist, then integer counting ranks them.

pair count=∑freq⋅adjacent occurrences\text{pair count}=\sum \text{freq}\cdot\text{adjacent occurrences}
Round one pair countsAdjacent pair counts are exact integers.Round one pair countsAdjacent pair counts are exact integers.round one countstie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h u gpug freq=2 symbols=p u gpair counts(h,u)=3(p,u)=2(u,g)=5

Round one counts

From the shown corpus, (h,u) has count 3, (u,g) has count 5, and (p,u) has count 2. The middle pair appears in both words, so its weighted total is larger than either word-specific starting pair.

(h,u)=3,(u,g)=5,(p,u)=2(h,u)=3,\quad (u,g)=5,\quad (p,u)=2
Round one pair countsAdjacent pair counts are exact integers.Round one pair countsAdjacent pair counts are exact integers.round one countstie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h u gpug freq=2 symbols=p u gpair counts(h,u)=3(p,u)=2(u,g)=5

Summary

The largest count is attached to (u,g). The next step applies the deterministic merge rule to that exact maximum. If counts tied, the stated tie-break would decide; here the maximum is strict.

max⁡{3,5,2}=5\max\{3,5,2\}=5
Round one pair countsAdjacent pair counts are exact integers.Round one pair countsAdjacent pair counts are exact integers.round one countstie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h u gpug freq=2 symbols=p u gpair counts(h,u)=3(p,u)=2(u,g)=5