Numerical gradients
Not met yet
A numerical gradient measures a slope instead of working it out: nudge the input a small step either side, and divide the change in the output by the distance between the two, (f(x + step) - f(x - step)) / (2 * step). This central difference is far more accurate than nudging one way only, because the curve's bend cancels out between the two sides. A step near 1e-5 is the usual choice: much bigger and the curve bends within the step, much smaller and rounding in the subtraction swamps the answer. With several inputs, nudge one at a time and hold the rest still, for one partial slope each. It costs two runs of the function per input, too slow to train with, so it is used to check slopes worked out by hand: it knows nothing of the algebra, so it can't make the same mistake.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks nudging only upwards, (f(x + step) - f(x)) / step, measures a slope as accurately as nudging both ways.
- Thinks the smaller the step, the more accurate the measured slope, without limit, when very small steps let rounding errors take over.
- Thinks measured slopes should replace worked-out ones in training, when they cost two runs of the function per input and are kept for checking.