Transformer blocks
Not met yet
A transformer is a stack of blocks, and each block does two things, each with a way round. First, multi-head attention: several heads, each with its own Wq, Wk and Wv, their outputs put side by side with np.concatenate(..., axis=1) and mixed by Wo. Then a feed-forward network, a ReLU layer and a second layer, used on every position's row on its own. Before each, layer norm brings every row to mean 0 and spread 1, then applies a gain and shift: batch norm turned round, so it needs no batch. After each, the result is added to what came in, X = X + ..., a residual connection that lets blame flow straight back. Position embeddings are added at the start, because attention alone ignores order.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks multi-head attention adds or averages the heads' outputs, rather than putting them side by side.
- Thinks np.concatenate with axis=1 stacks arrays as extra rows rather than extra columns.
- Thinks layer norm takes each column's mean across the batch, like batch norm, rather than each row's.
- Thinks a block's output replaces what came in, rather than being added to it.
- Thinks attention knows the order of the characters without position embeddings.