The Backward Pass

Same Weights, Different Gate

The same ReLU network can send a different backward signal on a second row because the forward ReLU gate state changed. This lesson compares the existing backprop row with the second exact forward row, keeping the weights fixed and recomputing the row-level gradients.

highlighted = computed this step

Same weights, two rows

Use the same tiny network on row A, x=(1,2), and row B, x=(2,1). The weights and biases do not change; only the row changes, so the ReLU gate state can change.

z2(A)=−1,z2(B)=1z_{2}(A)=-1,\quad z_{2}(B)=1
Same Weights, Different GateTwo exact row-level backward passes reuse the same parameters.Same Weights, Different GateTwo exact row-level backward passes reuse the same parameters.same weights, different ReLU gatesame parameters; the backward path follows each row's forward ReLU gatesrowxtargetz2gate2dyhatdz2dw21dw22db2Row A(1,2)3-10-20000Row B(2,1)21122422Row A closes branch 2z2=-1; ReLU'(z2)=0; dz2=-2*0=0second-branch gradients: dw21=0, dw22=0, db2=0Row B opens branch 2z2=1; ReLU'(z2)=1; dz2=2*1=2second-branch gradients: dw21=4, dw22=2, db2=2one exact row-level backward comparison only; same weights and biases; NOT training; NOT convergence; NOT learning; NOT generalization

Row A closes the second gate

For row A, z2=-1, so ReLU'(z2)=0. Multiplying by that gate gives dL/dz2=0, so the second-branch entries dw21, dw22, and db2 are all 0.

dLdz2=−2⋅0=0\frac{dL}{dz_2}=-2\cdot0=0
Same Weights, Different GateTwo exact row-level backward passes reuse the same parameters.Same Weights, Different GateTwo exact row-level backward passes reuse the same parameters.same weights, different ReLU gatesame parameters; the backward path follows each row's forward ReLU gatesrowxtargetz2gate2dyhatdz2dw21dw22db2Row A(1,2)3-10-20000Row B(2,1)21122422Row A closes branch 2z2=-1; ReLU'(z2)=0; dz2=-2*0=0second-branch gradients: dw21=0, dw22=0, db2=0Row B opens branch 2z2=1; ReLU'(z2)=1; dz2=2*1=2second-branch gradients: dw21=4, dw22=2, db2=2one exact row-level backward comparison only; same weights and biases; NOT training; NOT convergence; NOT learning; NOT generalization

Row B opens the second gate

For row B, z2=1, so ReLU'(z2)=1. The second branch now carries dL/dz2=2, giving dw21=4, dw22=2, and db2=2.

dw21=2⋅2=4,dw22=2⋅1=2dw_{21}=2\cdot2=4,\quad dw_{22}=2\cdot1=2
Same Weights, Different GateTwo exact row-level backward passes reuse the same parameters.Same Weights, Different GateTwo exact row-level backward passes reuse the same parameters.same weights, different ReLU gatesame parameters; the backward path follows each row's forward ReLU gatesrowxtargetz2gate2dyhatdz2dw21dw22db2Row A(1,2)3-10-20000Row B(2,1)21122422Row A closes branch 2z2=-1; ReLU'(z2)=0; dz2=-2*0=0second-branch gradients: dw21=0, dw22=0, db2=0Row B opens branch 2z2=1; ReLU'(z2)=1; dz2=2*1=2second-branch gradients: dw21=4, dw22=2, db2=2one exact row-level backward comparison only; same weights and biases; NOT training; NOT convergence; NOT learning; NOT generalization

Boundary

Backprop follows the actual forward gates for the row being differentiated. This is one exact row-level backward comparison only: not training, not convergence, not learning, and no generalization claim is made.

same weights; row-level gates choose the backward path\text{same weights; row-level gates choose the backward path}
Same Weights, Different GateTwo exact row-level backward passes reuse the same parameters.Same Weights, Different GateTwo exact row-level backward passes reuse the same parameters.same weights, different ReLU gatesame parameters; the backward path follows each row's forward ReLU gatesrowxtargetz2gate2dyhatdz2dw21dw22db2Row A(1,2)3-10-20000Row B(2,1)21122422Row A closes branch 2z2=-1; ReLU'(z2)=0; dz2=-2*0=0second-branch gradients: dw21=0, dw22=0, db2=0Row B opens branch 2z2=1; ReLU'(z2)=1; dz2=2*1=2second-branch gradients: dw21=4, dw22=2, db2=2one exact row-level backward comparison only; same weights and biases; NOT training; NOT convergence; NOT learning; NOT generalization