Scale Boundary

Counting Parameters Exactly

Dense-layer parameter counts are exact integer formulas. The toy network is a small instance of the same formula.

highlighted = computed this step

Dense-layer count

For a dense layer, each output has one weight for every input and one bias. The exact formula is output dimension times input dimension plus one.

params=dout(din+ 1)\text{params}=d_{\text{out}}(d_{\text{in}}+\,1)
Dense layer formulaThe toy layers recompute from d_in and d_out.toy MLP parameter countlayerd_ind_outweightsbiastotalhidden22426output21213total params=9exact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization

Recompute the toy

For the hidden layer, 2 times (2 plus 1) equals 6. For the output layer, 1 times (2 plus 1) equals 3. Together they equal 9.

2(2 + 1)=6,1(2 + 1)=32(2\,+\,1)=6,\quad 1(2\,+\,1)=3
Dense layer formulaThe toy layers recompute from d_in and d_out.toy MLP parameter countlayerd_ind_outweightsbiastotalhidden22426output21213total params=9exact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization

Summary

Parameter counts are exact integer arithmetic. The problem is not that the formula is mysterious; the problem is how quickly the counts grow.

exact formula; growing count\text{exact formula; growing count}
Dense layer formulaThe toy layers recompute from d_in and d_out.toy MLP parameter countlayerd_ind_outweightsbiastotalhidden22426output21213total params=9exact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization