Activation Cells Also Grow - Why Toys Can't Scale

Output activations are exact integer tables too. This lesson counts tokens times width for three displayed shapes only.

highlighted = computed this step

Cells are rows times width

After a layer runs, its output activations form a table. The exact count is tokens times width. For the toy output, tokens=4 and width=2, so the activation table has 8 cells.

\text{activation cells}=\text{tokens}\times\text{width}

Same tokens, wider table

Keeping tokens at 4 but using width=768 gives 4 times 768 = 3072 activation cells.

4\times768=3072

More tokens, same width

With tokens=128 and width=768, the same formula gives 128 times 768 = 98304 cells. This is a finite list of displayed shapes only, not a smooth curve or benchmark.

128\times768=98304

What this count is not

These are exact activation-cell counts for displayed shapes only. They are not memory bytes, not latency, not a full model run, not training, not learning, and not generalization.

\text{displayed activation cells only}