The Forward Pass

ReLU

ReLU turns a pre-activation into a hidden activation by a sign test. One positive value passes through and one negative value becomes zero.

highlighted = computed this step

The exact ReLU rule

ReLU is max of zero and the shown pre-activation. It is a sign test, not a sigmoid, so no transcendental value appears.

ReLU⁡(z)=max⁡(0,z)\operatorname{ReLU}(z)=\max(0,z)
ReLU activationsReLU is an exact sign test.ReLU activationsReLU is an exact sign test.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0

The two hidden activations

Since z1=2 is positive, h1=2. Since z2=-1 is negative, h2=0.

h1=2,h2=0h_1=2,\quad h_2=0
ReLU activationsReLU is an exact sign test.ReLU activationsReLU is an exact sign test.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0

Summary

The negative pre-activation is zeroed exactly. That is the whole nonlinearity in this forward pass.

z2<0⇒h2=0z_2<0\Rightarrow h_2=0
ReLU activationsReLU is an exact sign test.ReLU activationsReLU is an exact sign test.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0