The chain rule
Not met yet
When one quantity depends on a second, and the second on a third, their rates multiply: if the focus knob turns 5 notches for every millimetre the brass tube shrinks, and the tube shrinks 0.02 millimetres for every degree the night cools, the knob turns 0.1 notches per degree. For a function inside a function, dy/dx = dy/du * du/dx: the outer function's slope, taken at the inner function's value, times the inner function's slope. So the slope of (3 * x + 1) ** 2 is 2 * (3 * x + 1) * 3, and leaving off that last * 3 is the classic mistake. A longer chain works the same way, one multiplication per link. Backpropagation is this rule run backwards from the loss, link by link.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Takes the slope of a function of a function as the outer slope alone, forgetting to multiply by the inner function's slope.
- Thinks the rates along a chain add up, rather than multiply.
- Works out the outer function's slope at x itself, rather than at the inner function's value.