Paul's Python CoursePPC

Policy evaluation

Not met yet

The value of a state under a policy is how good it is to stand there: the return you can expect from there, following that policy. Each value depends on its neighbours: what the policy's action is worth there, its Q-value, is for every outcome the reward plus gamma times the value of where it leads (nothing, if the episode is over), weighted by its chance and added up. Policy evaluation starts every value at 0 and sweeps through the states, setting each from its neighbours, until the biggest change in a sweep is below a small threshold, theta. News of a reward travels back about one step per sweep.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks one sweep of policy evaluation carries the finish's value all the way back to the start, when news travels about one tile per sweep.
  • Thinks a tile's value is only the reward for its next step, leaving out gamma times the value of where it leads.
  • Thinks the finish has a value of its own equal to its reward, when the race is over there and its value is 0; the reward belongs to the step onto it.

Where it comes up