Wiring the Block

Layer Normalization

A Named Step

Layernorm introduces a square-root boundary through standard deviation. The block surface names that operation instead of pinning a decimal.

highlighted = computed this step

Layer normalization

Layernorm rescales a vector using its standard deviation. That standard deviation contains a square root, so the component is marked named.

layernorm uses variance\text{layernorm uses }\sqrt{\text{variance}}
Layernorm named stepLayernorm nodes are named because they use a square root.Layernorm named stepLayernorm nodes are named because they use a square root.transformer block wiringwiring is exact; named components carry no numeric outputinput hexactdisplayed vector+ positional encodingnamed (sin/cos)attentionnamed (softmax)residual addexactvector additionlayernormnamed (√)MLP-ReLUexactrational ReLU wiringresidual addexactvector additionlayernormnamed (√)outputexactresidual example: h=(2,3) + s=(1,-1) = (3,2)

A named step

The diagram does not pin a layernorm output as a decimal. It only marks the component as named with a square-root boundary.

layernorm=named ( )\text{layernorm}=\text{named }(\sqrt{\ })
Layernorm named stepLayernorm nodes are named because they use a square root.Layernorm named stepLayernorm nodes are named because they use a square root.transformer block wiringwiring is exact; named components carry no numeric outputinput hexactdisplayed vector+ positional encodingnamed (sin/cos)attentionnamed (softmax)residual addexactvector additionlayernormnamed (√)MLP-ReLUexactrational ReLU wiringresidual addexactvector additionlayernormnamed (√)outputexactresidual example: h=(2,3) + s=(1,-1) = (3,2)

Summary

The wiring into and out of layernorm is exact, but the layernorm computation itself is named. That is the boundary for this node.

exact wiring around a named node\text{exact wiring around a named node}
Layernorm named stepLayernorm nodes are named because they use a square root.Layernorm named stepLayernorm nodes are named because they use a square root.transformer block wiringwiring is exact; named components carry no numeric outputinput hexactdisplayed vector+ positional encodingnamed (sin/cos)attentionnamed (softmax)residual addexactvector additionlayernormnamed (√)MLP-ReLUexactrational ReLU wiringresidual addexactvector additionlayernormnamed (√)outputexactresidual example: h=(2,3) + s=(1,-1) = (3,2)