Words as Vectors

Words as Vectors

An embedding table gives words coordinate vectors. This book uses assigned toy coordinates so the arithmetic stays exact and inspectable.

highlighted = computed this step

Words as vectors

An embedding table assigns each word an exact coordinate vector. This toy table uses integer coordinates so every displayed quantity can be checked directly.

word↦(x1,x2)\text{word}\mapsto (x_1,x_2)
Assigned toy vectorsWords have exact integer coordinates.Assigned toy vectorsWords have exact integer coordinates.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)

Assigned toy coordinates

The coordinates are assigned for this toy example: cat=(3,1), kitten=(2,1), and car=(1,3).

cat=(3,1),kitten=(2,1),car=(1,3)\text{cat}=(3,1),\quad \text{kitten}=(2,1),\quad \text{car}=(1,3)
Assigned toy vectorsWords have exact integer coordinates.Assigned toy vectorsWords have exact integer coordinates.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)

Summary

The table and plot are two views of the same exact vector map. The next lesson computes dot products from these displayed integers.

displayed vectors→integer dot products\text{displayed vectors}\rightarrow\text{integer dot products}
Assigned toy vectorsWords have exact integer coordinates.Assigned toy vectorsWords have exact integer coordinates.assigned toy embedding tabledisplayed vectors are assigned toy coordinatescat=(3,1)kitten=(2,1)car=(1,3)exact word vectors2D toy word vectorsArrows from the origin to exact assigned integer coordinates.ckac=cat=(3,1)k=kitten=(2,1)a=car=(1,3)