Autograd
Not met yet
An autograd engine works out gradients for you. Every value remembers the values it was made from and leaves behind a small _backward function holding its operation's local slope. backward() puts every value in topological order, each after its parents, sets the answer's gradient to 1, and walks the order backwards so each value's blame is complete before it is passed on. Blame is added with +=, so a value used twice gets both shares, and so gradients must be set back to 0 before each training step.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks a value used twice in a calculation gets its gradient from only one of the uses.
- Confuses a value's gradient with its number, as if .grad held the result of the calculation.
- Thinks calling backward again starts the gradients afresh, when each call adds to what is already there.
- Thinks the backward functions can be called in any order, when each value's blame must be complete before it is passed on.
- Thinks putting a shared value on the backward order twice does no harm, when its blame is then passed back twice.