Residual Connections

Residual connections add a sublayer output back to its input. The example here is exact vector addition, recomputed from displayed integers.

highlighted = computed this step

A residual connection adds a sublayer output back to its input. The rule is exact vector addition: output equals sublayer of x plus x.

\text{out}=\text{sublayer}(x)+x

Exact residual add

Here the input vector is (2,3) and the sublayer vector is (1,-1). Their exact sum is (3,2).

(2,3)+(1,-1)=(3,2)

The residual add keeps a direct path through the block, and the add itself is exact. No transcendental operation appears in this step.

\text{residual add is exact}