The Forward Pass

Output and Loss

The output node is another exact linear combination. The squared error then compares that output with the shown target.

highlighted = computed this step

The output node

The output combines the hidden activations with v1=1, v2=1, and c=0. The result is yhat=2.

y^=1⋅2  +  1⋅0  +  0=2\hat y=1\cdot2\;+\;1\cdot0\;+\;0=2
Output and lossThe output and squared loss are exact.Output and lossThe output and squared loss are exact.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0yhat1*2 + 1*0 + 02L(yhat - 3)^21

Squared error

The target is y=3. Squared error gives L=1 exactly.

L=(y^−y)2=(2  −  3)2=1L=(\hat y-y)^2=(2\;-\;3)^2=1
Output and lossThe output and squared loss are exact.Output and lossThe output and squared loss are exact.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0yhat1*2 + 1*0 + 02L(yhat - 3)^21

Summary

The forward pass ends with one exact integer loss. No decimal approximation is introduced.

y^=2,L=1\hat y=2,\quad L=1
Output and lossThe output and squared loss are exact.Output and lossThe output and squared loss are exact.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0yhat1*2 + 1*0 + 02L(yhat - 3)^21