Returns and discounting
Not met yet
The return is what an episode is worth from a given moment: the rewards from then on, each multiplied by gamma (the Greek letter γ, a number from 0 to 1) once more than the one before, so r0 + gamma * r1 + gamma**2 * r2 + .... The first reward counts in full. With gamma near 0 only the next reward or two matter; near 1 the far future counts almost as much as now, so gamma can change which choice is best. Discounting also keeps the return of an endless episode finite. Working backwards from the last reward, total = reward + gamma * total gives the return without any powers.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Adds up the rewards without discounting the later ones.
- Discounts the first reward too, using gamma to the power 1 where it should be gamma to the power 0.
- Thinks the return is the reward at the end of the episode, leaving out the rewards on the way.
- Thinks discounting shrinks every return by the same amount, so it can never change which choice is better.
- Thinks a larger gamma discounts the future more heavily, when a gamma near 1 hardly discounts at all.
- Thinks a gamma of 0 makes every reward count for nothing, when the first reward still counts in full.