Paul's Python CoursePPC

Cross-validation

Not met yet

One validation split is the luck of the draw: which rows land in it changes the score. k-fold cross-validation cuts the training data into k folds, fits a fresh model on all but one fold, scores it on the fold left out, and lets every fold take a turn. The mean of the k scores is a steadier guess at how the model does on new data. cross_val_score(model, X, y, cv=5) gives back the five scores. It's for choosing between models and settings, it leaves no fitted model behind, and the test set stays out of it.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks cross-validation fits one model and scores it on every fold, rather than fitting a fresh model for each fold.
  • Thinks cross_val_score leaves a fitted model behind, ready to predict.
  • Thinks a single validation split gives as steady a score as the mean over several folds.
  • Thinks each fold's model is trained on all the data, rather than on every fold but the one it is scored on.

Where it comes up