Paul's Python CoursePPC

Double descent

Not met yet

Make a model more flexible and its test error usually falls, then rises as it starts to fit the noise: the classical U. With smallest-norm fits, that rise peaks at the interpolation threshold, where the model has just enough parameters to fit every training example, and then the error falls again as the model keeps growing, sometimes below the bottom of the U. Past the threshold, many sets of weights fit the training data exactly; choosing the one with the smallest norm is what brings the error back down. A ridge penalty flattens the peak. Belkin, Hsu, Ma and Mandal named it in 2019.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks test error keeps rising as a model gains parameters past the bottom of the U. With smallest-norm fits it peaks at the interpolation threshold and then falls.
  • Thinks every set of weights that fits the training data exactly is as good as any other. Which exact fit is chosen decides the test error.
  • Thinks the smallest model has the worst test error. Small models underfit, but the peak is at the interpolation threshold.

Where it comes up