Temporal difference learning
Not met yet
TD(0) learns after every step, without waiting for the episode to end. The TD target is the reward just received plus gamma times the next state's current estimate (just the reward, if the episode is over). The estimate moves a fraction alpha (the Greek letter α, a number from 0 to 1) of the way towards it: V[s] += alpha * (target - V[s]). Target minus the old estimate is the TD error. TD learns a guess from a guess, so news of a reward travels back one state per visit, and with a fixed alpha the estimates keep wobbling a little.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks a TD update sets the value to the TD target, ignoring the step size alpha.
- Leaves the discounted value of the next state out of the TD target, using the reward alone.
- Thinks TD learning must wait for the end of an episode before updating, as Monte Carlo does.
- Adds gamma times the value of a terminal state to the TD target, or lets a terminal state's value be updated, when nothing follows the end of an episode.
- Thinks one episode of TD(0) carries a reward back to every state visited, when each update only looks one step ahead.