Ensembles: forests and boosting
Not met yet
An ensemble combines many models into one answer. Averaging helps only when the models are wrong in different ways, so a random forest makes its trees differ: each grows deep on a bootstrap sample (as many rows as the data, drawn with replacement, so some repeat and about a third are left out), and each question may only consider a few random readings. Each tree overfits in its own way; the average cancels much of it. Gradient boosting grows trees one after another instead: start from the mean, fit a small tree to what's left over, y - prediction, add a learning rate's worth of its answer, and repeat. Small trees and a small rate keep it careful; a big rate and deep trees drive the training error to nothing and overfit.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks averaging many models that each make errors can't do better than one of them.
- Thinks averaging copies of one model, or models grown identically on the same data, improves on a single one.
- Thinks a bootstrap sample is the data reshuffled with every row exactly once, rather than drawn with replacement, so some rows repeat and others are left out.
- Thinks boosting adds each new tree's whole answer, rather than the learning rate times it.
- Thinks each tree in boosting learns the whole target on its own, as a forest's trees do, rather than fitting what the trees before it still got wrong.
- Thinks boosting can't overfit, so a bigger rate, deeper trees and more rounds always help because the training error keeps falling.