Backpropagation
Not met yet
Backpropagation works out how the loss changes with every weight by running the chain rule backwards. Do the forward pass and keep every value; then start at the loss and walk back, multiplying the blame arriving at each value by that link's local slope. A weight's gradient is the blame arriving times what went in. ReLU passes blame only where it fired, sigmoid scales it by P * (1 - P), and every gradient has the same shape as the thing it belongs to.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks blame passes back through a ReLU even where it cut its input to 0.
- Thinks blame passes back through a sigmoid unchanged, as if its slope were 1.
- Thinks the sigmoid's slope is a quarter everywhere, as it is at 0, so a saturated neuron still passes back plenty of blame.
- Thinks a weight's gradient can have a different shape from the weight, such as one row per example.