The same tiny network can be applied to a second input row. The weights and biases are reused; only the row and target change.
Same weights and biases
Both rows use the same network: hidden weights 1,1 with b1=-1; hidden weights 1,-1 with b2=0; output weights 1,1 with c=0.
same weights and biases for both rows \text{same weights and biases for both rows} same weights and biases for both rows Same Weights, Two Rows Two exact forward passes reuse the same parameters. Same Weights, Two Rows Two exact forward passes reuse the same parameters. same weights, two rows shared parameters: h1 weights [1,1], b1=-1; h2 weights [1,-1], b2=0; output weights [1,1], c=0 row x target z1 z2 h yhat L Row A (1,2) 3 2 -1 (2,0) 2 1 Row B (2,1) 2 2 1 (2,1) 3 1 Row B recompute z1 = 1*2 + 1*1 - 1 = 2 z2 = 1*2 - 1*1 + 0 = 1 h = (2,1); yhat=3; L=(3-2)^2=1 two independent exact forward passes; same weights and biases; NOT training; NOT learning; NOT generalization; NOT batch optimizer
Row B hidden sums
For Row B, x=(2, 1). The hidden sums are z1=2 + 1 - 1=2 and z2=2 - 1 + 0=1.
z 1 = 2 + 1 − 1 = 2 , z 2 = 1 z_1=2 + 1 - 1=2,\quad z_2=1 z 1 = 2 + 1 − 1 = 2 , z 2 = 1 Same Weights, Two Rows Two exact forward passes reuse the same parameters. Same Weights, Two Rows Two exact forward passes reuse the same parameters. same weights, two rows shared parameters: h1 weights [1,1], b1=-1; h2 weights [1,-1], b2=0; output weights [1,1], c=0 row x target z1 z2 h yhat L Row A (1,2) 3 2 -1 (2,0) 2 1 Row B (2,1) 2 2 1 (2,1) 3 1 Row B recompute z1 = 1*2 + 1*1 - 1 = 2 z2 = 1*2 - 1*1 + 0 = 1 h = (2,1); yhat=3; L=(3-2)^2=1 two independent exact forward passes; same weights and biases; NOT training; NOT learning; NOT generalization; NOT batch optimizer
Row B output and loss
ReLU gives h=(2,1). Then yhat=3 and with target 2 the squared loss is 1.
h = ( 2 , 1 ) , y ^ = 3 , L = ( 3 − 2 ) 2 = 1 h=(2,1),\quad \hat y=3,\quad L=(3 - 2)^2=1 h = ( 2 , 1 ) , y ^ = 3 , L = ( 3 − 2 ) 2 = 1 Same Weights, Two Rows Two exact forward passes reuse the same parameters. Same Weights, Two Rows Two exact forward passes reuse the same parameters. same weights, two rows shared parameters: h1 weights [1,1], b1=-1; h2 weights [1,-1], b2=0; output weights [1,1], c=0 row x target z1 z2 h yhat L Row A (1,2) 3 2 -1 (2,0) 2 1 Row B (2,1) 2 2 1 (2,1) 3 1 Row B recompute z1 = 1*2 + 1*1 - 1 = 2 z2 = 1*2 - 1*1 + 0 = 1 h = (2,1); yhat=3; L=(3-2)^2=1 two independent exact forward passes; same weights and biases; NOT training; NOT learning; NOT generalization; NOT batch optimizer
Two independent forward passes
Row A has yhat=2 and L=1. Row B has yhat=3 and L=1. This is two exact forward passes only: NOT training, NOT learning, NOT generalization, and NOT a batch optimizer.
same parameters, two exact rows \text{same parameters, two exact rows} same parameters, two exact rows Same Weights, Two Rows Two exact forward passes reuse the same parameters. Same Weights, Two Rows Two exact forward passes reuse the same parameters. same weights, two rows shared parameters: h1 weights [1,1], b1=-1; h2 weights [1,-1], b2=0; output weights [1,1], c=0 row x target z1 z2 h yhat L Row A (1,2) 3 2 -1 (2,0) 2 1 Row B (2,1) 2 2 1 (2,1) 3 1 Row B recompute z1 = 1*2 + 1*1 - 1 = 2 z2 = 1*2 - 1*1 + 0 = 1 h = (2,1); yhat=3; L=(3-2)^2=1 two independent exact forward passes; same weights and biases; NOT training; NOT learning; NOT generalization; NOT batch optimizer