SARSA
Not met yet
SARSA is Q-learning's on-policy cousin. Its update uses State, Action, Reward, next State and next Action: it chooses the next action first, exploring or not, and its target is the reward plus gamma times the Q-value of that action. So it learns what actions are worth if the agent carries on exactly as it is, exploration included. Near a hazard, where one exploring step is costly, SARSA learns to keep its distance; Q-learning learns the best route and walks it while still exploring. As epsilon shrinks towards 0, the two agree.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks SARSA's target uses the best next action, as Q-learning's does, rather than the next action actually chosen.
- Swaps the two methods, thinking Q-learning uses the next action actually taken and SARSA the best one.
- Thinks SARSA learns the values of the best policy while it explores, when it learns the values of the exploring policy it follows.