Words as Vectors

What Embeddings Are and Are Not

The finale states the honesty boundary. The book pins vector geometry, not claims about where real embeddings come from or what words mean.

highlighted = computed this step

What is exact

The exact register is the assigned vector table plus integer dot products. The nearest-by-dot result is kitten because the exact dot is 7 rather than 6.

cat⋅kitten=7>6=cat⋅car\text{cat}\cdot\text{kitten}=7>6=\text{cat}\cdot\text{car}
Embedding honesty boundaryExact vectors and named cosine boundary.Embedding honesty boundaryExact vectors and named cosine boundary.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact dot productstie-break: lexicographically smallest word among max dot productsvectors:cat=(3,1); kitten=(2,1); car=(1,3)pairdotcosine named boundarycat·kitten77/(√10·√5)cat·car66/(√10·√10)kitten·car55/(√5·√10)nearest to cat by dot=kitten (dot=7)√ is named; no decimal cosine is pinnedexact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)

What is not claimed

These are assigned toy coordinate vectors. This does not claim learned embeddings, does not claim meaning or semantics, and does not claim understanding.

not learned embeddings; not meaning; not understanding\text{not learned embeddings; not meaning; not understanding}
Embedding honesty boundaryExact vectors and named cosine boundary.Embedding honesty boundaryExact vectors and named cosine boundary.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact dot productstie-break: lexicographically smallest word among max dot productsvectors:cat=(3,1); kitten=(2,1); car=(1,3)pairdotcosine named boundarycat·kitten77/(√10·√5)cat·car66/(√10·√10)kitten·car55/(√5·√10)nearest to cat by dot=kitten (dot=7)√ is named; no decimal cosine is pinnedexact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)

Summary

These are assigned toy coordinate vectors with exact integer dot products. This pins the geometric mechanics; it does not claim learned embeddings, does not claim meaning or semantics, and does not claim understanding.

geometric mechanics on assigned toy vectors\text{geometric mechanics on assigned toy vectors}
Embedding honesty boundaryExact vectors and named cosine boundary.Embedding honesty boundaryExact vectors and named cosine boundary.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact dot productstie-break: lexicographically smallest word among max dot productsvectors:cat=(3,1); kitten=(2,1); car=(1,3)pairdotcosine named boundarycat·kitten77/(√10·√5)cat·car66/(√10·√10)kitten·car55/(√5·√10)nearest to cat by dot=kitten (dot=7)√ is named; no decimal cosine is pinnedexact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)