LSTM gates
Not met yet
An LSTM keeps a second memory, the cell c, beside its hidden state, and controls it with gates: numbers from 0 to 1, made by sigmoids, that multiply what passes through. Each step makes four weighted sums from the reading and the hidden state: the forget, input and output gates (sigmoid) and a candidate (tanh). Then c = forget * c + inp * candidate and h = out * np.tanh(c). A forget gate near 1 means keep. Because the cell is only multiplied by a gate and added to, never squashed every step, a memory, and the blame flowing back to it, can last far longer than in a plain recurrent network.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks a forget gate near 1 wipes the memory, when it means keep almost all of it.
- Thinks a plain recurrent memory, squashed through tanh and a weight each step, keeps its value as well as a gated cell does.
- Thinks an LSTM's cell and its hidden state are the same thing, rather than the hidden state being the output gate times tanh of the cell.