Paul's Python CoursePPC

Actor-critic and PPO's clip

Not met yet

Actor-critic learns after every move. The critic keeps a value for each state, learned by TD; its verdict on a move is the TD error, reward + gamma * value(after) - value(before), an estimate of how much better the move was than usual from there. The actor is a softmax policy that takes the REINFORCE step with that error in place of the whole return. PPO takes several steps on one batch of moves but caps the reward for change: it maximises the smaller of ratio * advantage and the ratio clipped to 1 ± epsilon, times the advantage.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks the critic's verdict on a move is the reward it earned, leaving out the change in the critic's value from one state to the next.
  • Thinks a move that earns a negative reward is always made less likely, when the actor judges it against what the critic expected from that state.
  • Thinks the critic picks the moves, when the actor chooses them and the critic only judges how they went.
  • Thinks PPO uses the clipped value whenever the ratio leaves the range, when the min takes it only if it is the less flattering of the two.
  • Leaves the clip out of PPO's objective, as if making a move likelier went on paying however far the policy moved from the one that raced.
  • Takes the bigger of PPO's plain and clipped objectives, letting a bad move off lightly, when PPO takes the smaller.

Where it comes up