Value iteration
Not met yet
Value iteration finds the best values directly. It sweeps like policy evaluation, but each state takes the value of its best action: the max over actions of the expected reward plus gamma times the value of where it leads. The max is over the actions, which the agent chooses; the outcomes of each action are averaged by their chances, because chance chooses those. When the values stop changing, the best policy is to take, in each state, the action with the biggest value. The best policy weighs risk as well as distance, so it is not always the shortest way.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Takes the max over the outcomes of an action as well as over the actions, as if the agent could choose where chance sends it.
- Thinks the best policy always takes the shortest way to the goal, whatever the risk on the way.
- Thinks heading away from a hazard is safe, forgetting that slips sideways can still land in it.