Paul's Python CoursePPC

Optimisers, mini-batches and the learning rate

Not met yet

An optimiser is the rule that turns gradients into steps. Plain descent moves each weight by the learning rate times its gradient: too small a rate crawls, too big a rate overshoots further each step. Momentum keeps a velocity per weight, beta * velocity + gradient, so steady slopes build up speed and zig-zags cancel. Adam gives each weight its own step from running averages of its gradient and its square; its first step moves every weight by about the rate. Mini-batches take a step for every few dozen shuffled rows instead of one per pass, each a noisy estimate, many more of them.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks a bigger learning rate always reaches the bottom sooner.
  • Thinks the loss must fall on every mini-batch step if training is working, when each batch's gradient is only a noisy estimate.
  • Thinks Adam's first step is bigger for a weight with a bigger gradient, as plain descent's is, when it moves every weight by about the rate.
  • Thinks Adam's step counter goes up once for each weight array, rather than once per step.
  • Thinks momentum's velocity starts again from 0 at every step, so momentum is the same as plain descent.
  • Thinks shuffling the readings and the answers separately keeps each answer with its own readings.

Where it comes up