Merges

What BPE Is and Is Not

The finale marks the honesty boundary for this deterministic tokenizer. Counts and merges are exact; claims about meaning are outside this render.

highlighted = computed this step

What is exact

The exact work is integer pair counting plus deterministic argmax merges. Here the merge counts are 5 for (u,g) and 3 for (h,ug).

integer counts and deterministic argmax\text{integer counts and deterministic argmax}
BPE honesty boundaryCounts and merges are exact for the shown corpus.BPE honesty boundaryCounts and merges are exact for the shown corpus.exact BPE merge sequencedeterministic merges: count pairs, choose max, tie-break lexicographicround 1source corpus:hug freq=3 symbols=h u gpug freq=2 symbols=p u gcounts: (h,u)=3, (p,u)=2, (u,g)=5merge (u,g)->ug count=5after: hug=h ug; pug=p uground 2source corpus:hug freq=3 symbols=h ugpug freq=2 symbols=p ugcounts: (h,ug)=3, (p,ug)=2merge (h,ug)->hug count=3after: hug=hug; pug=p ug

What is not claimed

This does not claim learned meaning and does not claim language understanding. It is a toy tokenization procedure with visible counts and merges.

not meaning; not language understanding\text{not meaning; not language understanding}
BPE honesty boundaryCounts and merges are exact for the shown corpus.BPE honesty boundaryCounts and merges are exact for the shown corpus.exact BPE merge sequencedeterministic merges: count pairs, choose max, tie-break lexicographicround 1source corpus:hug freq=3 symbols=h u gpug freq=2 symbols=p u gcounts: (h,u)=3, (p,u)=2, (u,g)=5merge (u,g)->ug count=5after: hug=h ug; pug=p uground 2source corpus:hug freq=3 symbols=h ugpug freq=2 symbols=p ugcounts: (h,ug)=3, (p,ug)=2merge (h,ug)->hug count=3after: hug=hug; pug=p ug

Summary

BPE is a deterministic integer procedure on this toy corpus: count pairs, choose an argmax, then merge. It does not claim learned meaning and does not claim language understanding; it pins the tokenization mechanics.

BPE mechanics on a toy corpus\text{BPE mechanics on a toy corpus}
BPE honesty boundaryCounts and merges are exact for the shown corpus.BPE honesty boundaryCounts and merges are exact for the shown corpus.exact BPE merge sequencedeterministic merges: count pairs, choose max, tie-break lexicographicround 1source corpus:hug freq=3 symbols=h u gpug freq=2 symbols=p u gcounts: (h,u)=3, (p,u)=2, (u,g)=5merge (u,g)->ug count=5after: hug=h ug; pug=p uground 2source corpus:hug freq=3 symbols=h ugpug freq=2 symbols=p ugcounts: (h,ug)=3, (p,ug)=2merge (h,ug)->hug count=3after: hug=hug; pug=p ug