Merges

Repeat the Procedure

After a merge, the corpus is re-segmented and the same rule runs again. The second round is computed from the post-merge rows, so the visible segmentation is the source for the next count.

highlighted = computed this step

Repeat the procedure

Round two starts from h ug and p ug. The same count-then-merge rule runs again on the new visible segmentation. Nothing about the procedure changes between rounds; only the symbols in each word have changed.

same deterministic rule\text{same deterministic rule}
Round two mergeThe procedure repeats on the new segmentation.Round two mergeThe procedure repeats on the new segmentation.round two mergetie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h ugpug freq=2 symbols=p ugpair counts(h,ug)=3(p,ug)=2chosen merge(h,ug)->hug count=3after mergehug freq=3 symbols=hugpug freq=2 symbols=p ug

Round two counts

The pair (h,ug) has count 3 and (p,ug) has count 2. The selected merge is (h,ug) into hug because its integer count is larger under the same deterministic rule.

(h,ug)=3,(p,ug)=2(h,ug)=3,\quad (p,ug)=2
Round two mergeThe procedure repeats on the new segmentation.Round two mergeThe procedure repeats on the new segmentation.round two mergetie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h ugpug freq=2 symbols=p ugpair counts(h,ug)=3(p,ug)=2chosen merge(h,ug)->hug count=3after mergehug freq=3 symbols=hugpug freq=2 symbols=p ug

Summary

The second merge is deterministic because the counts are exact and ordered. The sequence so far is (u,g) to ug, then (h,ug) to hug. The ordered list is part of the tokenizer state.

(u,g)→ug,(h,ug)→hug(u,g)\rightarrow ug,\quad (h,ug)\rightarrow hug
Round two mergeThe procedure repeats on the new segmentation.Round two mergeThe procedure repeats on the new segmentation.round two mergetie-break: lexicographically smallest pair among max countscurrent corpushug freq=3 symbols=h ugpug freq=2 symbols=p ugpair counts(h,ug)=3(p,ug)=2chosen merge(h,ug)->hug count=3after mergehug freq=3 symbols=hugpug freq=2 symbols=p ug