Paul's Python CoursePPC

Deep Q networks

Not met yet

Deep Q (DQN) swaps Q-learning's table for a network: a state's features go in, one Q-value per action comes out. Each memory trains only the output for the action that was taken, towards the target reward + gamma * max(frozen Q-values after), or the reward alone if that move ended the episode. Two tricks keep it steady: a replay buffer, so each step learns from a random batch of old and new memories rather than the last few moves in order, and a frozen copy of the network for the targets, refreshed every so many moves. Epsilon starts high and falls as it learns.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Adds gamma times the value of a terminal state to the TD target, or lets a terminal state's value be updated, when nothing follows the end of an episode.
  • Uses the next state's value for the action just taken in the Q-learning target, instead of the best action's value there.
  • Thinks the TD error should train every action's output for a tile, when only the move that was actually taken has a target.
  • Thinks replaying memories in the order they happened is best, when consecutive moves are so alike that a network trained on them in order forgets everything but the latest stretch.
  • Thinks assigning the weights to a second name makes a frozen copy, when both names share the same arrays and the copy changes with every training step.
  • Thinks the frozen copy of the network is made once and kept for ever, when it is refreshed from the trained network every so many moves.

Where it comes up