Checking gradients by measuring
Not met yet
To check a backward pass, measure each gradient instead: nudge one weight a hair up, then a hair down, and divide the change in the loss by the distance between them, 2 * step. Put the weight back before moving on. If np.allclose says the measured and worked-out gradients agree, backward is right. A falling loss proves nothing: a gradient that points roughly downhill can be wrong and still go down.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks a measured slope from a nudge up and a nudge down is divided by the step, not by twice the step.
- Thinks nudging each weight to measure a slope leaves the weights as they were, without putting each one back.
- Thinks that if the loss goes down during training, the gradients from backward must be right.