Scores, Masks, Softmax

The Attention Grid

The full grid combines scores, mask, and named softmax in one validated scene. It keeps exact integers, structural mask cells, and named weights visible at the same time.

highlighted = computed this step

The attention grid

The full grid shows the three pieces together: exact dot-product scores, the causal mask, and named row softmax. It is the compact view of attention's register split: integer scores and structural exclusions before the named boundary.

scores→mask→named softmax\text{scores}\rightarrow\text{mask}\rightarrow\text{named softmax}
Attention gridExact scores, exact mask, named row softmax.Attention gridExact scores, exact mask, named row softmax.exact scores, exact causal mask, named softmaxdisplayed Q,K integer vectors are the score source; causal mask keeps j<=iQ1=(1,0); Q2=(0,1); Q3=(1,1)K1=(1,0); K2=(0,1); K3=(1,1)score S_ij=Qi·Kj with causal maskK1K2K3Q1Q2Q31maskedmasked01masked112row softmax over unmasked scoresrow 1: scores [1]weights 1row 2: scores [0,1]weights e^0/(e^0+e^1), e^1/(e^0+e^1)row 3: scores [1,1,2]weights e^1/(e^1+e^1+e^2), e^1/(e^1+e^1+e^2), e^2/(e^1+e^1+e^2)softmax is named: no decimal attention weights are pinned

Reading the grid

The first row has weight 1 because it has only one allowed key. The other rows keep their exponential forms so no softmax float is shown. This is why the grid can be exact-or-named without a third category.

row 1 weight 1\text{row }1\text{ weight }1
Attention gridExact scores, exact mask, named row softmax.Attention gridExact scores, exact mask, named row softmax.exact scores, exact causal mask, named softmaxdisplayed Q,K integer vectors are the score source; causal mask keeps j<=iQ1=(1,0); Q2=(0,1); Q3=(1,1)K1=(1,0); K2=(0,1); K3=(1,1)score S_ij=Qi·Kj with causal maskK1K2K3Q1Q2Q31maskedmasked01masked112row softmax over unmasked scoresrow 1: scores [1]weights 1row 2: scores [0,1]weights e^0/(e^0+e^1), e^1/(e^0+e^1)row 3: scores [1,1,2]weights e^1/(e^1+e^1+e^2), e^1/(e^1+e^1+e^2), e^2/(e^1+e^1+e^2)softmax is named: no decimal attention weights are pinned

Summary

The diagram is self-contained: Q and K are visible, scores recompute from them, mask cells are structural, and softmax stays named. The trust boundary is enforced by the rendered sources.

displayed source→validated grid\text{displayed source}\rightarrow\text{validated grid}
Attention gridExact scores, exact mask, named row softmax.Attention gridExact scores, exact mask, named row softmax.exact scores, exact causal mask, named softmaxdisplayed Q,K integer vectors are the score source; causal mask keeps j<=iQ1=(1,0); Q2=(0,1); Q3=(1,1)K1=(1,0); K2=(0,1); K3=(1,1)score S_ij=Qi·Kj with causal maskK1K2K3Q1Q2Q31maskedmasked01masked112row softmax over unmasked scoresrow 1: scores [1]weights 1row 2: scores [0,1]weights e^0/(e^0+e^1), e^1/(e^0+e^1)row 3: scores [1,1,2]weights e^1/(e^1+e^1+e^2), e^1/(e^1+e^1+e^2), e^2/(e^1+e^1+e^2)softmax is named: no decimal attention weights are pinned