Paul's Python CoursePPC

Monte Carlo learning

Not met yet

Monte Carlo learns what states are worth from whole episodes, with no model of the world. When an episode ends, walk back through it: the return from each step is its reward plus gamma times the return from the step after. Each return is one sample of what that state is worth, and its value is the running average of its samples. Counting every visit or only the first in each episode both work. It can't learn until an episode is over.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Credits every state in an episode with the whole episode's return from the start, rather than the return from the moment the agent was in that state.
  • Discounts each reward by its distance from the start of the episode when working out a later state's return, instead of counting from that state.
  • Thinks Monte Carlo needs the transition probabilities, when it learns from sampled episodes alone.
  • Thinks Monte Carlo can update a state's value before the episode ends, when the return is not known until then.

Where it comes up