Words as Vectors

Words as Vectors

An embedding table gives words coordinate vectors. This book uses assigned toy coordinates so the arithmetic stays exact and inspectable, and every geometric claim is tied back to displayed integers.

highlighted = computed this step

Words as vectors

An embedding table assigns each word an exact coordinate vector. In this toy book the coordinates are integers, so the table is not a hidden source: every dot product, plotted point, and nearest-by-dot choice can be checked from the displayed vector entries.

word↦(x1,x2)\text{word}\mapsto (x_1,x_2)
Assigned toy vectorsWords have exact integer coordinates.Assigned toy vectorsWords have exact integer coordinates.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)

Assigned toy coordinates

The coordinates are assigned for this toy example: cat=(3,1), kitten=(2,1), and car=(1,3). The word labels are names for rows in the table; the arithmetic uses only the shown coordinates.

cat=(3,1),kitten=(2,1),car=(1,3)\text{cat}=(3,1),\quad \text{kitten}=(2,1),\quad \text{car}=(1,3)
Assigned toy vectorsWords have exact integer coordinates.Assigned toy vectorsWords have exact integer coordinates.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)

Summary

The table and plot are two views of the same exact vector map. The next lesson computes dot products from these displayed integers, so the geometric picture and the numeric table stay locked together.

displayed vectors→integer dot products\text{displayed vectors}\rightarrow\text{integer dot products}
Assigned toy vectorsWords have exact integer coordinates.Assigned toy vectorsWords have exact integer coordinates.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)