Scale Boundary

What We Computed by Hand

The exact neural-net books used a tiny parameter list. This lesson counts that list explicitly before the scale boundary appears.

highlighted = computed this step

The toy network

The hand-computed neural-net books used inputs=2, hidden units=2, and output units=1. Those sizes are small enough that every parameter can be named and checked.

din=2,dhidden=2,dout=1d_{\text{in}}=2,\quad d_{\text{hidden}}=2,\quad d_{\text{out}}=1
Toy MLP countExact parameter count for the toy network.toy MLP parameter countlayerd_ind_outweightsbiastotalhidden22426output21213total params=9exact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization

Nine parameters

The first dense layer has 4 weights and 2 biases, so it has 6 parameters. The output layer has 2 weights and 1 bias, so it has 3. Total: 9.

6 + 3=96\,+\,3=9
Toy MLP countExact parameter count for the toy network.toy MLP parameter countlayerd_ind_outweightsbiastotalhidden22426output21213total params=9exact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization

Summary

For the toy net, parameter enumeration is feasible. The whole parameter list fits in one exact table.

toy parameter count=9\text{toy parameter count}=9
Toy MLP countExact parameter count for the toy network.toy MLP parameter countlayerd_ind_outweightsbiastotalhidden22426output21213total params=9exact mechanisms on toys; scale blocks hand enumeration of real trained values; finite grid only;NOT learning; NOT generalization