Tokenization (BPE)

Merges

• Subword Tokens
• Counting Adjacent Pairs
• When Pair Counts Tie
• Merge the Most Frequent Pair
• Repeat the Procedure
• The Merge Sequence
• Apply the Merge List
• What BPE Is and Is Not