Attention is exact until a multi-entry softmax appears. The one-entry row pins to one; the two-entry row is named, marking the attention boundary.
Attention, exactly
The Q, K, and V maps are identity matrices, so the displayed vectors pass through unchanged. Position zero can attend only to itself. That makes the first attention row fully exact: one score, one allowed key, one exact weight.
Attention in the full pass Position zero exact; position one named. Attention in the full pass Position zero exact; position one named. tiny transformer exact-or-named forward discrete spine exact; softmax/layernorm sqrt become named only at the boundary input: a b; E[a]=(1,0), E[b]=(0,1), E[c]=(1,1); P0=(0,0), P1=(1,0) weights: Wq=Wk=Wv=I; MLP=I+ReLU+I; gamma=(1,1), beta=(0,0); unembed tied to E position 0: fully exact path x0=(1,0); Q0=K0=V0=(1,0) score S00=1; softmax=[1] exact attn0=(1,0); residual1=(2,0) ln1 mean=1; centered=(1,-1); var=1; std=1 ln1 output=(1,-1) MLP ReLU=(1,0); mlp=(1,0) residual2=(2,-1) ln2 mean=1/2; centered=(3/2,-3/2); var=9/4; std=3/2 ln2 output=(1,-1) logits: a=1, b=-1, c=0 argmax=a; output token=a position 1: named softmax boundary x1=(1,1); Q1=(1,1) K0=(1,0); K1=(1,1) scores=[1, 2] softmax=[e^1/(e^1+e^2), e^2/(e^1+e^2)] named after multi-entry softmax ordered pipeline tokens -> embed -> +pos -> attention -> +residual -> layernorm -> MLP -> +residual -> layernorm -> unembed -> logits -> argmax -> output token pos0 remains exact; pos1 stops at named softmax
One-entry softmax
Position zero has score 1 and a one-entry softmax, so the attention weight is exactly 1. Therefore attn zero is (1,0). This is the exact branch of attention inside the full pass.
S zero,zero = 1 , softmax = [ 1 ] S_{\text{zero,zero}}=1,\quad \operatorname{softmax}=[1] S zero,zero = 1 , softmax = [ 1 ] Attention in the full pass Position zero exact; position one named. Attention in the full pass Position zero exact; position one named. tiny transformer exact-or-named forward discrete spine exact; softmax/layernorm sqrt become named only at the boundary input: a b; E[a]=(1,0), E[b]=(0,1), E[c]=(1,1); P0=(0,0), P1=(1,0) weights: Wq=Wk=Wv=I; MLP=I+ReLU+I; gamma=(1,1), beta=(0,0); unembed tied to E position 0: fully exact path x0=(1,0); Q0=K0=V0=(1,0) score S00=1; softmax=[1] exact attn0=(1,0); residual1=(2,0) ln1 mean=1; centered=(1,-1); var=1; std=1 ln1 output=(1,-1) MLP ReLU=(1,0); mlp=(1,0) residual2=(2,-1) ln2 mean=1/2; centered=(3/2,-3/2); var=9/4; std=3/2 ln2 output=(1,-1) logits: a=1, b=-1, c=0 argmax=a; output token=a position 1: named softmax boundary x1=(1,1); Q1=(1,1) K0=(1,0); K1=(1,1) scores=[1, 2] softmax=[e^1/(e^1+e^2), e^2/(e^1+e^2)] named after multi-entry softmax ordered pipeline tokens -> embed -> +pos -> attention -> +residual -> layernorm -> MLP -> +residual -> layernorm -> unembed -> logits -> argmax -> output token pos0 remains exact; pos1 stops at named softmax
Weighting the value, term by term
This SHOWS the value-weighting with the shared walkthrough. Position zero attends one key, so its weight is exactly 1 and the output is one times V zero, the exact vector (1,0) = attn zero. The multi-key rows stay named.
output zero = 1 ⋅ V zero = ( 1 , 0 ) \text{output}_{\text{zero}}=1\cdot V_{\text{zero}}=(1,0) output zero = 1 ⋅ V zero = ( 1 , 0 ) Attention value-weighting at position 0 Position 0 attends one key, so its softmax weight is exactly 1 and the output is V0=(1,0). attention output = Σ weightⱼ · Vⱼ (per query row) V: V1=(1,0); V2=(0,3); V3=(1,1) Q: Q1=(1,0); Q2=(0,1); Q3=(1,1) K: K1=(1,0); K2=(0,1); K3=(1,1) query Q1 attends {K1} — single key: weight is EXACTLY 1 1·V1=(1,0) output1 = 1·V1 = (1,0) (EXACT) query Q2 attends {K1,K2} — multiple keys: NAMED weights weight·V1=(e^0/(e^0+e^1))·(1,0) ; weight·V2=(e^1/(e^0+e^1))·(0,3) output2 = Σ (e^0/(e^0+e^1))·(1,0) + (e^1/(e^0+e^1))·(0,3) (NAMED) query Q3 attends {K1,K2,K3} — multiple keys: NAMED weights weight·V1=(e^1/(e^1+e^1+e^2))·(1,0) ; weight·V2=(e^1/(e^1+e^1+e^2))·(0,3) ; weight·V3=(e^2/(e^1+e^1+e^2))·(1,1) output3 = Σ (e^1/(e^1+e^1+e^2))·(1,0) + (e^1/(e^1+e^1+e^2))·(0,3) + (e^2/(e^1+e^1+e^2))·(1,1) (NAMED) weights stay exact-or-named: no softmax weight is a decimal softmax weights are the NAMED boundary: row 1 attends one key so its weight is 1 EXACTLY and the output is V1; multi-key rows keep NAMED weights and NAMED outputs. NOT decimalized, NOT learning, NOT a claim the model attends to what matters
The named boundary
Position one has two unmasked scores, 1 and 2. Its softmax is named with exponentials, not decimalized.
softmax ( [ 1 , 2 ] ) = named \operatorname{softmax}([1,2])=\text{named} softmax ([ 1 , 2 ]) = named Attention in the full pass Position zero exact; position one named. Attention in the full pass Position zero exact; position one named. tiny transformer exact-or-named forward discrete spine exact; softmax/layernorm sqrt become named only at the boundary input: a b; E[a]=(1,0), E[b]=(0,1), E[c]=(1,1); P0=(0,0), P1=(1,0) weights: Wq=Wk=Wv=I; MLP=I+ReLU+I; gamma=(1,1), beta=(0,0); unembed tied to E position 0: fully exact path x0=(1,0); Q0=K0=V0=(1,0) score S00=1; softmax=[1] exact attn0=(1,0); residual1=(2,0) ln1 mean=1; centered=(1,-1); var=1; std=1 ln1 output=(1,-1) MLP ReLU=(1,0); mlp=(1,0) residual2=(2,-1) ln2 mean=1/2; centered=(3/2,-3/2); var=9/4; std=3/2 ln2 output=(1,-1) logits: a=1, b=-1, c=0 argmax=a; output token=a position 1: named softmax boundary x1=(1,1); Q1=(1,1) K0=(1,0); K1=(1,1) scores=[1, 2] softmax=[e^1/(e^1+e^2), e^2/(e^1+e^2)] named after multi-entry softmax ordered pipeline tokens -> embed -> +pos -> attention -> +residual -> layernorm -> MLP -> +residual -> layernorm -> unembed -> logits -> argmax -> output token pos0 remains exact; pos1 stops at named softmax