Paul's Python CoursePPC

Backpropagation

Not met yet

Backpropagation works out how the loss changes with every weight by running the chain rule backwards. Do the forward pass and keep every value; then start at the loss and walk back, multiplying the blame arriving at each value by that link's local slope. A weight's gradient is the blame arriving times what went in. ReLU passes blame only where it fired, sigmoid scales it by P * (1 - P), and every gradient has the same shape as the thing it belongs to.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks blame passes back through a ReLU even where it cut its input to 0.
  • Thinks blame passes back through a sigmoid unchanged, as if its slope were 1.
  • Thinks the sigmoid's slope is a quarter everywhere, as it is at 0, so a saturated neuron still passes back plenty of blame.
  • Thinks a weight's gradient can have a different shape from the weight, such as one row per example.

Where it comes up