The output is a weighted sum of value vectors. This lesson shows the value matrix V and the attention output: an exact integer vector for the single-key query, and named weighted sums for multi-key queries.
The value vectors
The output reads from a third matrix, the values. The displayed value vectors are integer pairs V1=(2,0), V2=(0,3), and V3=(1,1). Each attention output is a weighted sum of these vectors over the unmasked keys.
output i = ∑ j ≤ i softmax ( S i ) j V j \text{output}_i=\sum_{j\le i}\operatorname{softmax}(S_i)_j\,V_j output i = j ≤ i ∑ softmax ( S i ) j V j Weighting the values, term by term Exact single-key output and named multi-key outputs over the book's Q, K, V. attention output = Σ weightⱼ · Vⱼ (per query row) V: V1=(2,0); V2=(0,3); V3=(1,1) Q: Q1=(1,0); Q2=(0,1); Q3=(1,1) K: K1=(1,0); K2=(0,1); K3=(1,1) query Q1 attends {K1} — single key: weight is EXACTLY 1 1·V1=(2,0) output1 = 1·V1 = (2,0) (EXACT) query Q2 attends {K1,K2} — multiple keys: NAMED weights weight·V1=(e^0/(e^0+e^1))·(2,0) ; weight·V2=(e^1/(e^0+e^1))·(0,3) output2 = Σ (e^0/(e^0+e^1))·(2,0) + (e^1/(e^0+e^1))·(0,3) (NAMED) query Q3 attends {K1,K2,K3} — multiple keys: NAMED weights weight·V1=(e^1/(e^1+e^1+e^2))·(2,0) ; weight·V2=(e^1/(e^1+e^1+e^2))·(0,3) ; weight·V3=(e^2/(e^1+e^1+e^2))·(1,1) output3 = Σ (e^1/(e^1+e^1+e^2))·(2,0) + (e^1/(e^1+e^1+e^2))·(0,3) + (e^2/(e^1+e^1+e^2))·(1,1) (NAMED) weights stay exact-or-named: no softmax weight is a decimal softmax weights are the NAMED boundary: row 1 attends one key so its weight is 1 EXACTLY and the output is V1; multi-key rows keep NAMED weights and NAMED outputs. NOT decimalized, NOT learning, NOT a claim the model attends to what matters
The one exact output
Query 1 attends to exactly one key, so its weight is exactly 1 and its output is V1 unchanged: (2,0). This is fully exact because a one-entry softmax is one.
output 1 = 1 ⋅ V 1 = ( 2 , 0 ) \text{output}_1=1\cdot V_1=(2,0) output 1 = 1 ⋅ V 1 = ( 2 , 0 ) Weighting the values, term by term Exact single-key output and named multi-key outputs over the book's Q, K, V. attention output = Σ weightⱼ · Vⱼ (per query row) V: V1=(2,0); V2=(0,3); V3=(1,1) Q: Q1=(1,0); Q2=(0,1); Q3=(1,1) K: K1=(1,0); K2=(0,1); K3=(1,1) query Q1 attends {K1} — single key: weight is EXACTLY 1 1·V1=(2,0) output1 = 1·V1 = (2,0) (EXACT) query Q2 attends {K1,K2} — multiple keys: NAMED weights weight·V1=(e^0/(e^0+e^1))·(2,0) ; weight·V2=(e^1/(e^0+e^1))·(0,3) output2 = Σ (e^0/(e^0+e^1))·(2,0) + (e^1/(e^0+e^1))·(0,3) (NAMED) query Q3 attends {K1,K2,K3} — multiple keys: NAMED weights weight·V1=(e^1/(e^1+e^1+e^2))·(2,0) ; weight·V2=(e^1/(e^1+e^1+e^2))·(0,3) ; weight·V3=(e^2/(e^1+e^1+e^2))·(1,1) output3 = Σ (e^1/(e^1+e^1+e^2))·(2,0) + (e^1/(e^1+e^1+e^2))·(0,3) + (e^2/(e^1+e^1+e^2))·(1,1) (NAMED) weights stay exact-or-named: no softmax weight is a decimal softmax weights are the NAMED boundary: row 1 attends one key so its weight is 1 EXACTLY and the output is V1; multi-key rows keep NAMED weights and NAMED outputs. NOT decimalized, NOT learning, NOT a claim the model attends to what matters
The named outputs
Query 2 and query 3 attend to several keys, so their weights are named softmax symbols. Their outputs are named weighted sums of value vectors, not decimals.
output i = ∑ j e S i j ∑ k e S i k V j ( named ) \text{output}_i=\sum_j \frac{e^{S_{ij}}}{\sum_k e^{S_{ik}}}\,V_j\quad(\text{named}) output i = j ∑ ∑ k e S ik e S ij V j ( named ) Weighting the values, term by term Exact single-key output and named multi-key outputs over the book's Q, K, V. attention output = Σ weightⱼ · Vⱼ (per query row) V: V1=(2,0); V2=(0,3); V3=(1,1) Q: Q1=(1,0); Q2=(0,1); Q3=(1,1) K: K1=(1,0); K2=(0,1); K3=(1,1) query Q1 attends {K1} — single key: weight is EXACTLY 1 1·V1=(2,0) output1 = 1·V1 = (2,0) (EXACT) query Q2 attends {K1,K2} — multiple keys: NAMED weights weight·V1=(e^0/(e^0+e^1))·(2,0) ; weight·V2=(e^1/(e^0+e^1))·(0,3) output2 = Σ (e^0/(e^0+e^1))·(2,0) + (e^1/(e^0+e^1))·(0,3) (NAMED) query Q3 attends {K1,K2,K3} — multiple keys: NAMED weights weight·V1=(e^1/(e^1+e^1+e^2))·(2,0) ; weight·V2=(e^1/(e^1+e^1+e^2))·(0,3) ; weight·V3=(e^2/(e^1+e^1+e^2))·(1,1) output3 = Σ (e^1/(e^1+e^1+e^2))·(2,0) + (e^1/(e^1+e^1+e^2))·(0,3) + (e^2/(e^1+e^1+e^2))·(1,1) (NAMED) weights stay exact-or-named: no softmax weight is a decimal softmax weights are the NAMED boundary: row 1 attends one key so its weight is 1 EXACTLY and the output is V1; multi-key rows keep NAMED weights and NAMED outputs. NOT decimalized, NOT learning, NOT a claim the model attends to what matters
Assigned exact weights
To show the multi-key blend without crossing the softmax boundary, assign exact weights 1/4, 1/4, and 1/2 to V1, V2, and V3. These assigned weights sum to 1 exactly, and the output recomputes to (1,5/4). This illustrates how attention blends values; it is not a computed softmax output.
assigned 1 / 4 V 1 + 1 / 4 V 2 + 1 / 2 V 3 = ( 1 , 5 / 4 ) \text{assigned }1/4V_1\;+\;1/4V_2\;+\;1/2V_3=(1,5/4) assigned 1/4 V 1 + 1/4 V 2 + 1/2 V 3 = ( 1 , 5/4 ) Assigned exact weights for value blending Assigned rational weights illustrate the weighted sum without computing softmax. assigned exact weights, not computed softmax assigned weights (not softmax): w1=1/4; w2=1/4; w3=1/2 weights sum = 1/4 + 1/4 + 1/2 = 1 (EXACT) V: V1=(2,0); V2=(0,3); V3=(1,1) 1/4·V1=1/4·(2,0)=(1/2,0) ; 1/4·V2=1/4·(0,3)=(0,3/4) ; 1/2·V3=1/2·(1,1)=(1/2,1/2) assigned output = Σ wⱼ·Vⱼ = (1,5/4) (EXACT) assigned weights illustrate the blend; they are NOT computed softmax output assigned weights illustrate how attention blends values; they are NOT computed softmax output. The softmax that assigns real weights is the NAMED boundary above. The assigned weights sum to 1 EXACTLY; NOT decimalized, NOT learning, NOT a claim the model attends to what matters
Summary
The value-weighting is exact-or-named with no third option: the single-key output is an exact integer vector, and the multi-key outputs stay named.
exact output for one key; named outputs otherwise \text{exact output for one key; named outputs otherwise} exact output for one key; named outputs otherwise Weighting the values, term by term Exact single-key output and named multi-key outputs over the book's Q, K, V. attention output = Σ weightⱼ · Vⱼ (per query row) V: V1=(2,0); V2=(0,3); V3=(1,1) Q: Q1=(1,0); Q2=(0,1); Q3=(1,1) K: K1=(1,0); K2=(0,1); K3=(1,1) query Q1 attends {K1} — single key: weight is EXACTLY 1 1·V1=(2,0) output1 = 1·V1 = (2,0) (EXACT) query Q2 attends {K1,K2} — multiple keys: NAMED weights weight·V1=(e^0/(e^0+e^1))·(2,0) ; weight·V2=(e^1/(e^0+e^1))·(0,3) output2 = Σ (e^0/(e^0+e^1))·(2,0) + (e^1/(e^0+e^1))·(0,3) (NAMED) query Q3 attends {K1,K2,K3} — multiple keys: NAMED weights weight·V1=(e^1/(e^1+e^1+e^2))·(2,0) ; weight·V2=(e^1/(e^1+e^1+e^2))·(0,3) ; weight·V3=(e^2/(e^1+e^1+e^2))·(1,1) output3 = Σ (e^1/(e^1+e^1+e^2))·(2,0) + (e^1/(e^1+e^1+e^2))·(0,3) + (e^2/(e^1+e^1+e^2))·(1,1) (NAMED) weights stay exact-or-named: no softmax weight is a decimal softmax weights are the NAMED boundary: row 1 attends one key so its weight is 1 EXACTLY and the output is V1; multi-key rows keep NAMED weights and NAMED outputs. NOT decimalized, NOT learning, NOT a claim the model attends to what matters