Scaled dot-product attention
Not met yet
In attention, each position makes a query, a key and a value. A query's score against each key is their dot product divided by the square root of the key size, so Q @ K.T / np.sqrt(d) scores every query against every key at once, one row per query. Softmax along each row turns the scores into weights that add up to 1, and weights @ V mixes the values: a little of every value, more of the best matches. In self-attention, Q, K and V are all made from the same rows of X. A causal mask sets the scores of later positions to minus infinity before softmax, so their weights are exactly 0 and nothing looks ahead; a score of 0 would still get a share.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks attention takes the value of the single best-matching key, rather than a softmax-weighted mix of every value.
- Thinks attention's output is the plain average of the values, ignoring the weights.
- Thinks setting a masked score to 0 gives it a weight of 0, when softmax gives e to the 0, which is 1.
- Thinks np.where with a mask leaves the scores as they were.
- Thinks attention's softmax runs down each column, across the queries, rather than along each row, across the keys.
- Thinks the scores are divided by the key size itself, rather than by its square root.
- Thinks attention knows the order of the characters without position embeddings.