Scale Boundary

Counting One Forward Layer

Operation counts are exact integer rows too. This lesson counts scalar terms for one dense layer pass, not a full model run.

highlighted = computed this step

Toy hidden layer

For one toy hidden layer pass, inputs=2 and units=2. Add one bias term per unit: 2 bias adds. The scalar terms are 2 times 2 plus 2 = 6.

2×2 + 2=62\times2\,+\,2=6
Counting one forward layerExact scalar term counts for one dense layer pass.one forward layer scalar termslayer passinputsunitsbias addsinput*unit + biastoy hidden forward2226real dense forward768768768590592one layer pass only; integer terms, not timingexact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization; one layer pass only; NOT a full model run; NOT training; NOTlatency; NOT hardware speed; NOT general performance

Real dense layer

For the real dense row, inputs=768 and units=768. The same count is 768 times 768 plus 768 = 590592.

768×768 + 768=590592768\times768\,+\,768=590592
Counting one forward layerExact scalar term counts for one dense layer pass.one forward layer scalar termslayer passinputsunitsbias addsinput*unit + biastoy hidden forward2226real dense forward768768768590592one layer pass only; integer terms, not timingexact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization; one layer pass only; NOT a full model run; NOT training; NOTlatency; NOT hardware speed; NOT general performance

What this count means

This counts one layer pass only. It is not a full model run, not latency, not hardware speed, and not a general performance claim. The arithmetic is exact; the boundary is enumeration and scale.

one layer pass only; exact integer rows\text{one layer pass only; exact integer rows}
Counting one forward layerExact scalar term counts for one dense layer pass.one forward layer scalar termslayer passinputsunitsbias addsinput*unit + biastoy hidden forward2226real dense forward768768768590592one layer pass only; integer terms, not timingexact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization; one layer pass only; NOT a full model run; NOT training; NOTlatency; NOT hardware speed; NOT general performance