Paul's Python CoursePPC

Recurrent networks

Not met yet

A recurrent neural network reads a sequence one item at a time and keeps a hidden state between items. Each step mixes the new reading with the old state, h = np.tanh(x * Wx + h @ Wh + b), and reads its answer off the new state, guess = h @ Wy + by. The same weights are used at every step, the state starts as zeros, and it is carried from one step to the next, never reset inside the loop. A guess made after reading t is about reading t + 1, so mark it against that. Training runs backpropagation through time: blame flows back step by step, multiplied by each step's slopes, so it fades (or explodes) over long stretches.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks a recurrent network starts each step with an empty memory, so it only ever sees the current reading.
  • Thinks a recurrent network's memory keeps old readings at full strength, rather than fading them through its memory weights.
  • Thinks a recurrent network has different weights for each step of the sequence, rather than the same weights reused.
  • Thinks the guess made at step t should be compared with the reading at step t, rather than the next one.
  • Thinks blame passed back through time arrives at earlier steps undiminished.
  • Thinks backpropagation through time only passes blame back one step.
  • Thinks the factors blame passes through, step after step, add up rather than multiply.

Where it comes up