The Forward Pass

ReLU

ReLU turns a pre-activation into a hidden activation by a sign test. One positive value passes through and one negative value becomes zero.

highlighted = computed this step

The exact ReLU rule

ReLU is max of zero and the shown pre-activation. It is a sign test, not a sigmoid, so no transcendental value appears.

ReLU⁡(z)=max⁡(0,z)\operatorname{ReLU}(z)=\max(0,z)
ReLU activationsReLU is an exact sign test.ReLU activationsReLU is an exact sign test.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0exact ReLU gate guide0ReLU(z)=max(0,z)zmax(0,z)-2(-2,0)-1(-1,0)0(0,0)1(1,1)2(2,2)gate rule: z<=0 -> 0, z>0 -> zsample checkzmax(0,z)gate-20clamp 0-10clamp 000clamp 011pass z22pass zgate rule: z<=0 -> 0, z>0 -> z

The two hidden activations

Since z1=2 is positive, h1=2. Since z2=-1 is negative, h2=0.

h1=2,h2=0h_1=2,\quad h_2=0
ReLU activationsReLU is an exact sign test.ReLU activationsReLU is an exact sign test.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0exact ReLU gate guide0ReLU(z)=max(0,z)zmax(0,z)-2(-2,0)-1(-1,0)0(0,0)1(1,1)2(2,2)gate rule: z<=0 -> 0, z>0 -> zsample checkzmax(0,z)gate-20clamp 0-10clamp 000clamp 011pass z22pass zgate rule: z<=0 -> 0, z>0 -> z

Summary

The negative pre-activation is zeroed exactly. That is the whole nonlinearity in this forward pass.

z2<0⇒h2=0z_2<0\Rightarrow h_2=0
ReLU activationsReLU is an exact sign test.ReLU activationsReLU is an exact sign test.forward pass (all exact)quantityrulevaluez11*1 + 1*2 - 12h1ReLU(z1)2z21*1 - 1*2 + 0-1h2ReLU(z2)0exact ReLU gate guide0ReLU(z)=max(0,z)zmax(0,z)-2(-2,0)-1(-1,0)0(0,0)1(1,1)2(2,2)gate rule: z<=0 -> 0, z>0 -> zsample checkzmax(0,z)gate-20clamp 0-10clamp 000clamp 011pass z22pass zgate rule: z<=0 -> 0, z>0 -> z