Byte-pair encoding starts from character symbols and builds larger tokens by merging adjacent pairs. This first lesson shows the toy corpus, its integer frequencies, and the current segmentation that all later counts must come from.
highlighted = computed this step
Subword tokens
BPE starts from character symbols and repeatedly merges the most frequent adjacent symbol pair. That is how it builds subword tokens: pieces larger than characters but smaller than a whole vocabulary of every word. The source here is a tiny corpus with integer frequencies, so every later merge is reproducible from visible rows.
start from characters
The shown corpus
The word hug has frequency 3 and symbols h u g. The word pug has frequency 2 and symbols p u g. The frequencies matter because each word contributes its adjacent pairs once per displayed occurrence, not just once per spelling.
hug: hug freq 3,pug: pug freq 2
Summary
The first register is the displayed corpus itself: word, frequency, and current segmentation. The next step counts adjacent pairs from these exact rows, so there is no hidden model state behind the merge choice.