Regularisation: weight decay and early stopping
Not met yet
A regulariser gives up a little fit on the training data to fit unseen data better, usually by stopping the model from bending itself round every noisy point. Weight decay adds decay * W to each weight matrix's gradient before the step, so every weight shrinks by a share of itself each step unless the loss pushes back; biases are left alone. Early stopping watches the validation loss every epoch, keeps a copy of the weights whenever it reaches a new low, stops after patience epochs without one, and puts the best weights back. Regularising usually raises the training loss. Judge it, and choose its strength, by the validation loss.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks weight decay only moves a weight when the loss's gradient for it is non-zero.
- Thinks weight decay takes the same fixed amount off a weight every step, rather than a share of the weight.
- Thinks weight decay switches unhelpful weights off at once, setting them to exactly 0.
- Thinks weight decay should shrink the biases as well as the weights.
- Thinks early stopping should watch the training loss, which keeps improving for ever.
- Thinks early stopping hands back the weights from the epoch it stopped at, rather than the best epoch's.
- Thinks early stopping stops the first time the validation loss rises, rather than waiting out its patience.
- Thinks early stopping compares each validation loss with the previous epoch's, rather than with the best so far.
- Thinks a regulariser should make the training loss lower, when it usually makes it higher in exchange for a lower validation loss.
- Thinks params.copy() copies the arrays inside the dict, so later in-place updates won't reach the copy.