Scale Boundary

A Real Configuration

Real-sized dimensions still produce exact integer counts. The numbers are easy to count and hard to enumerate value by value.

highlighted = computed this step

Embedding table count

A real-sized embedding table can be counted exactly. With vocab size 50000 and model dimension 768, the count is 38400000.

50000×768=3840000050000\times768=38400000
A real configurationLarge exact integer counts.real-size integer countscomponentleftrightbiascountembedding table50000768038400000dense layer768768768590592counts exact; trained values not enumerated by handexact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization

Dense layer count

A dense layer from 768 to 768 has 768 times 768 weights plus 768 biases, for 590592 parameters.

768×768 + 768=590592768\times768\,+\,768=590592
A real configurationLarge exact integer counts.real-size integer countscomponentleftrightbiascountembedding table50000768038400000dense layer768768768590592counts exact; trained values not enumerated by handexact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization

Summary

These counts are exact, but they are already far beyond hand enumeration of individual trained values. Scale is the boundary being named here.

exact count; too many values to enumerate by hand\text{exact count; too many values to enumerate by hand}
A real configurationLarge exact integer counts.real-size integer countscomponentleftrightbiascountembedding table50000768038400000dense layer768768768590592counts exact; trained values not enumerated by handexact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization