Tokenization (BPE)
A lightweight execution-visualization book.
Merges
Subword Tokens
Counting Adjacent Pairs
Merge the Most Frequent Pair
Repeat the Procedure
The Merge Sequence
What BPE Is and Is Not