TinyStories, Traced

Tokens — Still Exact

The tokenizer and greedy output produce integer IDs. Those IDs are the exact discrete spine inside the captured run.

highlighted = computed this step

Tokens are still exact

The prompt becomes token IDs [7454, 2402, 257, 640]. These are integer tokenizer outputs, not rounded magnitudes.

[7454,2402,257,640][7454,2402,257,640]
Exact token spinePrompt and greedy output IDs are exact integers.Exact token spinePrompt and greedy output IDs are exact integers.discrete token spineprompt: 'Once upon a time'prompt token IDs = [7454, 2402, 257, 640]output token IDs = [7454, 2402, 257, 640, 11, 612, 373]greedy exact: step 0: id 11 text ',' | step 1: id 612 text ' there' | step 2: id 373 text ' was'real output text = 'Once upon a time, there was'

Greedy output IDs

The greedy run appends IDs 11, 612, and 373. The resulting text is the captured string shown in the panel.

append 11,612,373\text{append }11,612,373
Exact token spinePrompt and greedy output IDs are exact integers.Exact token spinePrompt and greedy output IDs are exact integers.discrete token spineprompt: 'Once upon a time'prompt token IDs = [7454, 2402, 257, 640]output token IDs = [7454, 2402, 257, 640, 11, 612, 373]greedy exact: step 0: id 11 text ',' | step 1: id 612 text ' there' | step 2: id 373 text ' was'real output text = 'Once upon a time, there was'

Summary

This is the same discrete-token spine as the tokenization book, now from a real tokenizer and real greedy run. The integer IDs remain bit-exact.

token IDs are exact integers\text{token IDs are exact integers}
Exact token spinePrompt and greedy output IDs are exact integers.Exact token spinePrompt and greedy output IDs are exact integers.discrete token spineprompt: 'Once upon a time'prompt token IDs = [7454, 2402, 257, 640]output token IDs = [7454, 2402, 257, 640, 11, 612, 373]greedy exact: step 0: id 11 text ',' | step 1: id 612 text ' there' | step 2: id 373 text ' was'real output text = 'Once upon a time, there was'