The final normalized vector unembeds to exact logits. Greedy argmax then selects the next token exactly, preserving the discrete spine through output.
Final layernorm
The second layernorm has variance 9/4 and std 3/2. Its exact output is (1,-1).
LN two = ( 1 , − 1 ) \operatorname{LN}_{\text{two}}=(1,-1) LN two = ( 1 , − 1 ) The next token Exact final layernorm, logits, and argmax. The next token Exact final layernorm, logits, and argmax. tiny transformer exact-or-named forward discrete spine exact; softmax/layernorm sqrt become named only at the boundary input: a b; E[a]=(1,0), E[b]=(0,1), E[c]=(1,1); P0=(0,0), P1=(1,0) weights: Wq=Wk=Wv=I; MLP=I+ReLU+I; gamma=(1,1), beta=(0,0); unembed tied to E position 0: fully exact path x0=(1,0); Q0=K0=V0=(1,0) score S00=1; softmax=[1] exact attn0=(1,0); residual1=(2,0) ln1 mean=1; centered=(1,-1); var=1; std=1 ln1 output=(1,-1) MLP ReLU=(1,0); mlp=(1,0) residual2=(2,-1) ln2 mean=1/2; centered=(3/2,-3/2); var=9/4; std=3/2 ln2 output=(1,-1) logits: a=1, b=-1, c=0 argmax=a; output token=a position 1: named softmax boundary x1=(1,1); Q1=(1,1) K0=(1,0); K1=(1,1) scores=[1, 2] softmax=[e^1/(e^1+e^2), e^2/(e^1+e^2)] named after multi-entry softmax ordered pipeline tokens -> embed -> +pos -> attention -> +residual -> layernorm -> MLP -> +residual -> layernorm -> unembed -> logits -> argmax -> output token pos0 remains exact; pos1 stops at named softmax
Unembed dot product, term by term
This SHOWS the tied unembed for token a with the shared dot-product walkthrough. The final normalized vector (1,-1) dotted with E[a]=(1,0) sums to the exact logit 1. The unembed is just dot products against the same token vectors.
ℓ a = LN two ⋅ E [ a ] = 1 \ell_a=\operatorname{LN}_{\text{two}}\cdot E[a]=1 ℓ a = LN two ⋅ E [ a ] = 1 Tied unembed logit for token a Exact integer dot product of the final normalized vector with a tied token embedding. logit(a) = LN2 · E[a] (one scalar) A c1 c2 1 -1 · x entry 1 0 products 1·1=1 Σ=1 + -1·0=0 Σ=1 result 1 u·v = 1 one exact unembed dot product; does not claim learned meaning; does not claim understanding
Tied unembed
The tied unembed dots that vector with the token embeddings. The logits are a=1, b=-1, c=0. These integer logits are the exact source for the final greedy decision.
ℓ a = 1 , ℓ b = − 1 , ℓ c = 0 \ell_a=1,\ell_b=-1,\ell_c=0 ℓ a = 1 , ℓ b = − 1 , ℓ c = 0 The next token Exact final layernorm, logits, and argmax. The next token Exact final layernorm, logits, and argmax. tiny transformer exact-or-named forward discrete spine exact; softmax/layernorm sqrt become named only at the boundary input: a b; E[a]=(1,0), E[b]=(0,1), E[c]=(1,1); P0=(0,0), P1=(1,0) weights: Wq=Wk=Wv=I; MLP=I+ReLU+I; gamma=(1,1), beta=(0,0); unembed tied to E position 0: fully exact path x0=(1,0); Q0=K0=V0=(1,0) score S00=1; softmax=[1] exact attn0=(1,0); residual1=(2,0) ln1 mean=1; centered=(1,-1); var=1; std=1 ln1 output=(1,-1) MLP ReLU=(1,0); mlp=(1,0) residual2=(2,-1) ln2 mean=1/2; centered=(3/2,-3/2); var=9/4; std=3/2 ln2 output=(1,-1) logits: a=1, b=-1, c=0 argmax=a; output token=a position 1: named softmax boundary x1=(1,1); Q1=(1,1) K0=(1,0); K1=(1,1) scores=[1, 2] softmax=[e^1/(e^1+e^2), e^2/(e^1+e^2)] named after multi-entry softmax ordered pipeline tokens -> embed -> +pos -> attention -> +residual -> layernorm -> MLP -> +residual -> layernorm -> unembed -> logits -> argmax -> output token pos0 remains exact; pos1 stops at named softmax
The next token
Greedy decoding takes the largest logit with lowest-index tie-break. The largest logit is a, so the output token is a. The final decision is discrete and exact even though other positions may have named softmax magnitudes.
argmax { a : 1 , b : − 1 , c : 0 } = a \operatorname{argmax}\{a:1,b:-1,c:0\}=a argmax { a : 1 , b : − 1 , c : 0 } = a The next token Exact final layernorm, logits, and argmax. The next token Exact final layernorm, logits, and argmax. tiny transformer exact-or-named forward discrete spine exact; softmax/layernorm sqrt become named only at the boundary input: a b; E[a]=(1,0), E[b]=(0,1), E[c]=(1,1); P0=(0,0), P1=(1,0) weights: Wq=Wk=Wv=I; MLP=I+ReLU+I; gamma=(1,1), beta=(0,0); unembed tied to E position 0: fully exact path x0=(1,0); Q0=K0=V0=(1,0) score S00=1; softmax=[1] exact attn0=(1,0); residual1=(2,0) ln1 mean=1; centered=(1,-1); var=1; std=1 ln1 output=(1,-1) MLP ReLU=(1,0); mlp=(1,0) residual2=(2,-1) ln2 mean=1/2; centered=(3/2,-3/2); var=9/4; std=3/2 ln2 output=(1,-1) logits: a=1, b=-1, c=0 argmax=a; output token=a position 1: named softmax boundary x1=(1,1); Q1=(1,1) K0=(1,0); K1=(1,1) scores=[1, 2] softmax=[e^1/(e^1+e^2), e^2/(e^1+e^2)] named after multi-entry softmax ordered pipeline tokens -> embed -> +pos -> attention -> +residual -> layernorm -> MLP -> +residual -> layernorm -> unembed -> logits -> argmax -> output token pos0 remains exact; pos1 stops at named softmax