Paul's Python CoursePPC

Gradient descent

Not met yet

Gradient descent finds the bottom of a loss by walking downhill, like a walker in fog who can only feel the slope underfoot. Each step works out the slope of the loss for every dial, then moves each dial against its slope by the learning rate times the slope: w = w - rate * dw. Steep ground gives long strides, and the strides shrink near the bottom on their own. Too small a rate crawls; too big a rate overshoots and climbs further every step, and nothing stops it except someone watching the loss.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks a bigger learning rate always reaches the bottom sooner.
  • Thinks gradient descent stops with an error when the learning rate is too big, rather than quietly walking further and further uphill.
  • Thinks gradient descent adds the slope, walking in the direction the gradient points, which is uphill.
  • Thinks every step of gradient descent moves the same distance, whatever the slope.
  • Thinks the learning rate is learnt along with the weights, rather than chosen by the person training.

Where it comes up