Policy gradients and REINFORCE
Not met yet
A policy gradient method learns the choosing directly. Preferences for each move go through a softmax to give chances, so every move keeps some chance and the agent explores by itself. REINFORCE steps the preferences by the return from each move onwards times the gradient of the log of that move's chance: for a softmax, 1 for the move made minus the chances. Taking a baseline, such as the usual return from that state, off every return leaves the long-run average step the same but makes each single step far steadier.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks a REINFORCE step always makes the move taken more likely, when a return below the baseline, or below zero with no baseline, makes it less likely.
- Thinks subtracting a baseline from every return can't change any single update, when it can turn a step against a move into a step towards it; only the long-run average stays the same.
- Thinks any baseline steadies the estimates, or a bigger one more so, when only a baseline close to the usual return from that state does.
- Thinks a softmax policy always takes the move with the biggest preference, when it takes every move with some chance.
- Thinks the REINFORCE step changes only the preference of the move taken, when the minus-chances part moves every move's preference on that state.