Decision trees
Not met yet
A decision tree forecasts with a chain of yes-or-no questions, each of the form "is this reading at most this threshold?", ending in a leaf that gives the answer. To grow one, measure how mixed a pile is, with Gini impurity (1 minus the sum of each label's share squared: 0 for a pure pile, 0.5 for half and half) or entropy. Try every reading, with thresholds halfway between neighbouring values, and keep the question whose two sides have the lowest Gini, each weighted by its size. Then grow each side the same way, by recursion, until a pile is pure, no question helps, or the tree reaches its maximum depth. With no limit, a tree learns its training data by heart, down to single odd nights; choose the depth on validation data.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks Gini impurity is the share of one label in a pile, rather than how mixed the labels are.
- Thinks Gini impurity is the share of the rarer label, rather than 1 minus the sum of each label's share squared.
- Thinks a question's score is the two sides' impurities added together, without weighting each by its size, which favours splitting off one odd case.
- Thinks a deeper tree always forecasts better, because it gets more of its training data right.
- Thinks each recursive call knows how deep it is without being told, so the depth needn't be passed on one higher.
- Thinks a tree only asks a question if its two answers lead to different labels, rather than whenever it makes the piles less mixed.