The Whole Pipeline

All Three Unembed Logits

The final normalized vector is compared with each tied token vector. This lesson spells out all three exact dot products before the existing next-token lesson makes the greedy choice.

highlighted = computed this step

Token a logit

Tied unembed compares the final normalized vector (1,-1) with the same token vectors used as embeddings. For token a, E[a]=(1,0), so the dot product gives logit a=1.

ℓa=(1,−1)⋅(1,0)=1\ell_a=(1,-1)\cdot(1,0)=1
Tied unembed logit for token aExact integer dot product of the final normalized vector with a tied token embedding.logit(a) = LN2 · E[a] (one scalar)Ac1c21-1·xentry10products1·1=1Σ=1+-1·0=0Σ=1result 1u·v = 1one exact unembed dot product; does not claim learned meaning; does not claimunderstanding

Token b logit

For token b, the tied vector is E[b]=(0,1). The same final vector dotted with E[b] gives logit b=-1.

ℓb=(1,−1)⋅(0,1)=−1\ell_b=(1,-1)\cdot(0,1)=-1
Tied unembed logit for token bExact integer dot product of the final normalized vector with a tied token embedding.logit(b) = LN2 · E[b] (one scalar)Ac1c21-1·xentry01products1·0=0Σ=0+-1·1=-1Σ=-1result -1u·v = -1one exact unembed dot product; does not claim learned meaning; does not claimunderstanding

Token c logit

For token c, E[c]=(1,1). The two terms cancel, so logit c=0.

ℓc=(1,−1)⋅(1,1)=0\ell_c=(1,-1)\cdot(1,1)=0
Tied unembed logit for token cExact integer dot product of the final normalized vector with a tied token embedding.logit(c) = LN2 · E[c] (one scalar)Ac1c21-1·xentry11products1·1=1Σ=1+-1·1=-1Σ=0result 0u·v = 0one exact unembed dot product; does not claim learned meaning; does not claimunderstanding

Compare the three logits

The three exact logits are a=1, b=-1, and c=0. The existing next-token lesson uses these integers for the final greedy choice. This step is one exact toy unembed calculation only: not training, not convergence, not learning, not meaning, not understanding, and no generalization claim is made.

ℓa=1,ℓb=−1,ℓc=0\ell_a=1,\quad \ell_b=-1,\quad \ell_c=0
All three unembed logitsExact tied-unembed logits before greedy choice.All three unembed logitsExact tied-unembed logits before greedy choice.tiny transformer exact-or-named forwarddiscrete spine exact; softmax/layernorm sqrt become named only at the boundaryinput: a b; E[a]=(1,0), E[b]=(0,1), E[c]=(1,1); P0=(0,0), P1=(1,0)weights: Wq=Wk=Wv=I; MLP=I+ReLU+I; gamma=(1,1), beta=(0,0); unembed tied to Eposition 0: fully exact pathx0=(1,0); Q0=K0=V0=(1,0)score S00=1; softmax=[1] exactattn0=(1,0); residual1=(2,0)ln1 mean=1; centered=(1,-1); var=1; std=1ln1 output=(1,-1)MLP ReLU=(1,0); mlp=(1,0)residual2=(2,-1)ln2 mean=1/2; centered=(3/2,-3/2); var=9/4; std=3/2ln2 output=(1,-1)logits: a=1, b=-1, c=0argmax=a; output token=aposition 1: named softmax boundaryx1=(1,1); Q1=(1,1)K0=(1,0); K1=(1,1)scores=[1, 2]softmax=[e^1/(e^1+e^2), e^2/(e^1+e^2)]named after multi-entry softmaxordered pipelinetokens -> embed -> +pos -> attention -> +residual-> layernorm -> MLP -> +residual -> layernorm-> unembed -> logits -> argmax -> output tokenpos0 remains exact; pos1 stops at named softmax