The Forward Pass

A Tiny Network

A neural net can be small enough to audit by hand. This opening lesson fixes the two-input, two-hidden-unit, one-output structure before the arithmetic is expanded.

highlighted = computed this step

The shown network

The diagram fixes a tiny network with 2 inputs, 2 ReLU hidden units, and 1 output. The point is that every arithmetic step fits on the page.

x∈Q2→h∈Q2→y^∈Qx\in\mathbb{Q}^2\rightarrow h\in\mathbb{Q}^2\rightarrow \hat y\in\mathbb{Q}
Tiny ReLU MLP structureNetwork structure and parameters are visible.Tiny ReLU MLP structureNetwork structure and parameters are visible.w11=1w12=1w21=1w22=-1ReLUReLUv1=1v2=1target=3x1inputx2inputz1weighted sumz2weighted sumh1ReLUh2ReLUyhatlinear outputLsquared lossb1=-1b2=0c=0z=0 convention: ReLU'(0)=0structure: 2 inputs -> 2 ReLU hidden units -> 1 output; about 10 params, not 100B

Small enough to audit

This is about 10 parameters, not 100B. The small scale is intentional: the graph can show all weights, biases, and node roles.

about 10 parameters, not 100B\text{about }10\text{ parameters, not }100\text{B}
Tiny ReLU MLP structureNetwork structure and parameters are visible.Tiny ReLU MLP structureNetwork structure and parameters are visible.w11=1w12=1w21=1w22=-1ReLUReLUv1=1v2=1target=3x1inputx2inputz1weighted sumz2weighted sumh1ReLUh2ReLUyhatlinear outputLsquared lossb1=-1b2=0c=0z=0 convention: ReLU'(0)=0structure: 2 inputs -> 2 ReLU hidden units -> 1 output; about 10 params, not 100B

Summary

The structure is fixed before any arithmetic is performed. The next lesson computes the two hidden weighted sums from the shown inputs and biases.

structure first, arithmetic next\text{structure first, arithmetic next}
Tiny ReLU MLP structureNetwork structure and parameters are visible.Tiny ReLU MLP structureNetwork structure and parameters are visible.w11=1w12=1w21=1w22=-1ReLUReLUv1=1v2=1target=3x1inputx2inputz1weighted sumz2weighted sumh1ReLUh2ReLUyhatlinear outputLsquared lossb1=-1b2=0c=0z=0 convention: ReLU'(0)=0structure: 2 inputs -> 2 ReLU hidden units -> 1 output; about 10 params, not 100B