Paul's Python CoursePPC

Q-learning

Not met yet

A Q-value, Q[state][action], is what an action is worth in a state if you play your best afterwards. With Q-values the best action is simply the biggest one, so no model of the world is needed to choose. Q-learning is TD learning for Q-values: after each step, the target is the reward plus gamma times the best Q-value in the next state (just the reward if the episode is over), and Q[state][action] moves a fraction alpha of the way towards it. Exploration, usually epsilon-greedy, gathers the experience; because the target always assumes the best next action, Q-learning learns the best policy's values while following an exploring one (off-policy).

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Uses the next state's value for the action just taken in the Q-learning target, instead of the best action's value there.
  • Averages the next state's action values in the Q-learning target, instead of taking the biggest.
  • Thinks choosing the best action from Q-values needs the transition probabilities, as choosing from state values does.
  • Thinks Q-learning's values include the cost of its own exploring, when its target always assumes the best action next.

Where it comes up