Paul's Python CoursePPC

Scaled dot-product attention

Not met yet

In attention, each position makes a query, a key and a value. A query's score against each key is their dot product divided by the square root of the key size, so Q @ K.T / np.sqrt(d) scores every query against every key at once, one row per query. Softmax along each row turns the scores into weights that add up to 1, and weights @ V mixes the values: a little of every value, more of the best matches. In self-attention, Q, K and V are all made from the same rows of X. A causal mask sets the scores of later positions to minus infinity before softmax, so their weights are exactly 0 and nothing looks ahead; a score of 0 would still get a share.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks attention takes the value of the single best-matching key, rather than a softmax-weighted mix of every value.
  • Thinks attention's output is the plain average of the values, ignoring the weights.
  • Thinks setting a masked score to 0 gives it a weight of 0, when softmax gives e to the 0, which is 1.
  • Thinks np.where with a mask leaves the scores as they were.
  • Thinks attention's softmax runs down each column, across the queries, rather than along each row, across the keys.
  • Thinks the scores are divided by the key size itself, rather than by its square root.
  • Thinks attention knows the order of the characters without position embeddings.

Where it comes up