A multi-key value blend can be shown with assigned rational weights. This lesson keeps the blend exact while stating that these weights are not computed softmax.
Choose assigned weights
Use assigned rational weights for this toy blend: 1/4 on V1, 1/4 on V2, and 1/2 on V3. They sum to 1 exactly. These are assigned weights for the lesson, not computed softmax weights.
assigned 1 / 4 + 1 / 4 + 1 / 2 = 1 \text{assigned }1/4\;+\;1/4\;+\;1/2\;=\;1 assigned 1/4 + 1/4 + 1/2 = 1 Assigned Weights Are Not Softmax Assigned rational weights blend values without computing softmax. assigned exact weights, not computed softmax assigned weights (not softmax): w1=1/4; w2=1/4; w3=1/2 weights sum = 1/4 + 1/4 + 1/2 = 1 (EXACT) V: V1=(2,0); V2=(0,3); V3=(1,1) 1/4·V1=1/4·(2,0)=(1/2,0) ; 1/4·V2=1/4·(0,3)=(0,3/4) ; 1/2·V3=1/2·(1,1)=(1/2,1/2) assigned output = Σ wⱼ·Vⱼ = (1,5/4) (EXACT) assigned weights illustrate the blend; they are NOT computed softmax output assigned weights illustrate how attention blends values; they are NOT computed softmax output. The softmax that assigns real weights is the NAMED boundary above. The assigned weights sum to 1 EXACTLY; NOT decimalized, NOT learning, NOT a claim the model attends to what matters
Multiply each value vector
The value vectors are V1=(2,0), V2=(0,3), and V3=(1,1). Multiplying by the assigned weights gives (1/2,0), (0,3/4), and (1/2,1/2).
1 / 4 V 1 = ( 1 / 2 , 0 ) , 1 / 4 V 2 = ( 0 , 3 / 4 ) , 1 / 2 V 3 = ( 1 / 2 , 1 / 2 ) 1/4V_1=(1/2,0),\quad 1/4V_2=(0,3/4),\quad 1/2V_3=(1/2,1/2) 1/4 V 1 = ( 1/2 , 0 ) , 1/4 V 2 = ( 0 , 3/4 ) , 1/2 V 3 = ( 1/2 , 1/2 ) Assigned Weights Are Not Softmax Assigned rational weights blend values without computing softmax. assigned exact weights, not computed softmax assigned weights (not softmax): w1=1/4; w2=1/4; w3=1/2 weights sum = 1/4 + 1/4 + 1/2 = 1 (EXACT) V: V1=(2,0); V2=(0,3); V3=(1,1) 1/4·V1=1/4·(2,0)=(1/2,0) ; 1/4·V2=1/4·(0,3)=(0,3/4) ; 1/2·V3=1/2·(1,1)=(1/2,1/2) assigned output = Σ wⱼ·Vⱼ = (1,5/4) (EXACT) assigned weights illustrate the blend; they are NOT computed softmax output assigned weights illustrate how attention blends values; they are NOT computed softmax output. The softmax that assigns real weights is the NAMED boundary above. The assigned weights sum to 1 EXACTLY; NOT decimalized, NOT learning, NOT a claim the model attends to what matters
Add the component vectors
Add x-components: 1/2 plus 0 plus 1/2 equals 1. Add y-components: 0 plus 3/4 plus 1/2 equals 5/4. The assigned blend output is (1,5/4).
( 1 / 2 , 0 ) + ( 0 , 3 / 4 ) + ( 1 / 2 , 1 / 2 ) = ( 1 , 5 / 4 ) (1/2,0)+(0,3/4)+(1/2,1/2)=(1,5/4) ( 1/2 , 0 ) + ( 0 , 3/4 ) + ( 1/2 , 1/2 ) = ( 1 , 5/4 ) Assigned Weights Are Not Softmax Assigned rational weights blend values without computing softmax. assigned exact weights, not computed softmax assigned weights (not softmax): w1=1/4; w2=1/4; w3=1/2 weights sum = 1/4 + 1/4 + 1/2 = 1 (EXACT) V: V1=(2,0); V2=(0,3); V3=(1,1) 1/4·V1=1/4·(2,0)=(1/2,0) ; 1/4·V2=1/4·(0,3)=(0,3/4) ; 1/2·V3=1/2·(1,1)=(1/2,1/2) assigned output = Σ wⱼ·Vⱼ = (1,5/4) (EXACT) assigned weights illustrate the blend; they are NOT computed softmax output assigned weights illustrate how attention blends values; they are NOT computed softmax output. The softmax that assigns real weights is the NAMED boundary above. The assigned weights sum to 1 EXACTLY; NOT decimalized, NOT learning, NOT a claim the model attends to what matters
Boundary
This is only an illustration of value blending with assigned rational weights. It does not compute softmax, does not claim learning, does not claim meaning, and does not claim what the model should attend to.
assigned blend only; softmax remains named \text{assigned blend only; softmax remains named} assigned blend only; softmax remains named Assigned Weights Are Not Softmax Assigned rational weights blend values without computing softmax. assigned exact weights, not computed softmax assigned weights (not softmax): w1=1/4; w2=1/4; w3=1/2 weights sum = 1/4 + 1/4 + 1/2 = 1 (EXACT) V: V1=(2,0); V2=(0,3); V3=(1,1) 1/4·V1=1/4·(2,0)=(1/2,0) ; 1/4·V2=1/4·(0,3)=(0,3/4) ; 1/2·V3=1/2·(1,1)=(1/2,1/2) assigned output = Σ wⱼ·Vⱼ = (1,5/4) (EXACT) assigned weights illustrate the blend; they are NOT computed softmax output assigned weights illustrate how attention blends values; they are NOT computed softmax output. The softmax that assigns real weights is the NAMED boundary above. The assigned weights sum to 1 EXACTLY; NOT decimalized, NOT learning, NOT a claim the model attends to what matters