Paul's Python CoursePPC

One-hot rows and embeddings

Not met yet

A one-hot row stands for a single token: zeros, with a 1 at the token's id. Multiplying it by a matrix E picks out one row, so one_hot(ids) @ E is the same as E[ids], only slower. That row is the token's embedding: a short list of numbers learnt like any other weight. Training moves tokens that keep similar company, such as digits that are all followed by spaces and commas, towards embeddings that point the same way. Compare embeddings by direction: divide each by its length, then take @.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks a one-hot row times a matrix adds up all the matrix's rows.
  • Thinks the 1 in a one-hot row at index i picks row i counting from 1.
  • Thinks a one-hot row can't be multiplied by an embedding matrix because the shapes don't line up.
  • Thinks looking up E[ids] gives different numbers from one_hot(ids) @ E.
  • Thinks a model groups embeddings by what the tokens mean, rather than by the company they keep in the training text.

Where it comes up