Wiring the Block

Layer Normalization

A Named Step

Layernorm introduces a square-root boundary through standard deviation. The block surface names that operation instead of pinning a decimal, while preserving the exact wiring around it.

highlighted = computed this step

Layer normalization

The residual add is exact first: (2,3) plus (1,-1) equals (3,2). Layernorm then uses the mean and variance of that residual vector.

(2,3)+(1,−1)=(3,2)(2,3)+(1,-1)=(3,2)
Residual add then layernormResidual stats are exact; layernorm sqrt/divide is named.Residual add then layernormResidual stats are exact; layernorm sqrt/divide is named.residual add then named layernormresidual arithmetic is exact; layernorm divide-by-std is namedresidual add(2,3) + (1,-1)(3,2)mean(3 + 2)/25/2centered vector(3,2) - 5/2(1/2,-1/2)variance((1/2)^2 + (-1/2)^2)/21/4layernorm: subtract mean, then divide by sqrt(1/4)sqrt/divide named boundary; no pinned layernorm output

A named step

The exact mean is 5/2 and the exact variance is 1/4. The square root and divide step is named, and the diagram does not pin layernorm output decimals.

μ=5/2,var⁡=1/4,var⁡ is named\mu=5/2,\quad \operatorname{var}=1/4,\quad \sqrt{\operatorname{var}}\text{ is named}
Residual add then layernormResidual stats are exact; layernorm sqrt/divide is named.Residual add then layernormResidual stats are exact; layernorm sqrt/divide is named.residual add then named layernormresidual arithmetic is exact; layernorm divide-by-std is namedresidual add(2,3) + (1,-1)(3,2)mean(3 + 2)/25/2centered vector(3,2) - 5/2(1/2,-1/2)variance((1/2)^2 + (-1/2)^2)/21/4layernorm: subtract mean, then divide by sqrt(1/4)sqrt/divide named boundary; no pinned layernorm output

Summary

Residual addition, mean, centered vector, and variance are exact here. The layernorm output remains unnamed because dividing by the square root belongs to the named boundary.

exact residual stats around a named sqrt/divide step\text{exact residual stats around a named sqrt/divide step}
Residual add then layernormResidual stats are exact; layernorm sqrt/divide is named.Residual add then layernormResidual stats are exact; layernorm sqrt/divide is named.residual add then named layernormresidual arithmetic is exact; layernorm divide-by-std is namedresidual add(2,3) + (1,-1)(3,2)mean(3 + 2)/25/2centered vector(3,2) - 5/2(1/2,-1/2)variance((1/2)^2 + (-1/2)^2)/21/4layernorm: subtract mean, then divide by sqrt(1/4)sqrt/divide named boundary; no pinned layernorm output