Paul's Python CoursePPC

Ensembles: forests and boosting

Not met yet

An ensemble combines many models into one answer. Averaging helps only when the models are wrong in different ways, so a random forest makes its trees differ: each grows deep on a bootstrap sample (as many rows as the data, drawn with replacement, so some repeat and about a third are left out), and each question may only consider a few random readings. Each tree overfits in its own way; the average cancels much of it. Gradient boosting grows trees one after another instead: start from the mean, fit a small tree to what's left over, y - prediction, add a learning rate's worth of its answer, and repeat. Small trees and a small rate keep it careful; a big rate and deep trees drive the training error to nothing and overfit.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks averaging many models that each make errors can't do better than one of them.
  • Thinks averaging copies of one model, or models grown identically on the same data, improves on a single one.
  • Thinks a bootstrap sample is the data reshuffled with every row exactly once, rather than drawn with replacement, so some rows repeat and others are left out.
  • Thinks boosting adds each new tree's whole answer, rather than the learning rate times it.
  • Thinks each tree in boosting learns the whole target on its own, as a forest's trees do, rather than fitting what the trees before it still got wrong.
  • Thinks boosting can't overfit, so a bigger rate, deeper trees and more rounds always help because the training error keeps falling.

Where it comes up