Paul's Python CoursePPC

LSTM gates

Not met yet

An LSTM keeps a second memory, the cell c, beside its hidden state, and controls it with gates: numbers from 0 to 1, made by sigmoids, that multiply what passes through. Each step makes four weighted sums from the reading and the hidden state: the forget, input and output gates (sigmoid) and a candidate (tanh). Then c = forget * c + inp * candidate and h = out * np.tanh(c). A forget gate near 1 means keep. Because the cell is only multiplied by a gate and added to, never squashed every step, a memory, and the blame flowing back to it, can last far longer than in a plain recurrent network.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks a forget gate near 1 wipes the memory, when it means keep almost all of it.
  • Thinks a plain recurrent memory, squashed through tanh and a weight each step, keeps its value as well as a gated cell does.
  • Thinks an LSTM's cell and its hidden state are the same thing, rather than the hidden state being the output gate times tanh of the cell.

Where it comes up