Merges

Merge the Most Frequent Pair

BPE chooses the pair with the largest count and merges it into a new token. A stated lexicographic tie-break makes the rule deterministic even when counts tie, so the procedure is reproducible.

highlighted = computed this step

Merge the most frequent pair

The argmax pair is (u,g) with count 5. The tie-break is lexicographically smallest pair, though this round has a strict maximum. This means the merge is a rule result, not a judgment call.

argmax⁡=(u,g),count=5\operatorname{argmax}=(u,g),\quad \text{count}=5
Round one mergeThe most frequent pair is merged deterministically.Round one mergeThe most frequent pair is merged deterministically.round one mergetie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h u gpug freq=2 symbols=p u gpair counts(h,u)=3(p,u)=2(u,g)=5chosen merge(u,g)->ug count=5after mergehug freq=3 symbols=h ugpug freq=2 symbols=p ug

The new token

Merging (u,g) creates the token ug. The segmentations become h ug with frequency 3 and p ug with frequency 2. Every occurrence of the adjacent pair is replaced in the current corpus before the next count table is built.

(u,g)→ug(u,g)\rightarrow ug
Round one mergeThe most frequent pair is merged deterministically.Round one mergeThe most frequent pair is merged deterministically.round one mergetie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h u gpug freq=2 symbols=p u gpair counts(h,u)=3(p,u)=2(u,g)=5chosen merge(u,g)->ug count=5after mergehug freq=3 symbols=h ugpug freq=2 symbols=p ug

Summary

After one merge, the displayed corpus has changed. The next round counts adjacent pairs in that new segmentation, which is why BPE is iterative: each merge changes the future counting surface.

h ug,p ugh\ ug,\quad p\ ug
Round one mergeThe most frequent pair is merged deterministically.Round one mergeThe most frequent pair is merged deterministically.round one mergetie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h u gpug freq=2 symbols=p u gpair counts(h,u)=3(p,u)=2(u,g)=5chosen merge(u,g)->ug count=5after mergehug freq=3 symbols=h ugpug freq=2 symbols=p ug