Paul's Python CoursePPC

Training, validation and test sets

Not met yet

A model's error on the data it learnt from flatters it: it has seen the answers. To find out how it does on new data, hold some back. Shuffle first, so every part looks alike, and shuffle x and y with one order so each reading keeps its answer. Choose between models on a validation set; keep a test set locked away until the very end, look at it once, and report that number.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks a low error on the training data shows the model will do well on data it has not seen.
  • Thinks it is fair to choose settings by trying them on the test set, and then report the test error as honest.
  • Thinks taking the first rows for training and the last for testing is always fair, however the rows are ordered.
  • Thinks x and y can be shuffled separately when splitting, rather than with the same order.

Where it comes up