ROC curves and AUC
Not met yet
A forecast that gives chances can be turned into decisions at any threshold, and each threshold gives one point: the false positive rate (the share of quiet nights warned) along the bottom, the true positive rate (recall) up the side. Trying every distinct chance as the threshold, from the highest down, traces the ROC curve from (0, 0), warning on nothing, to (1, 1), warning on everything. The area under it, the AUC, is the chance that a random real case got a higher forecast than a random quiet one: 1 is perfect, 0.5 is a coin. It sums up every threshold at once, so it measures how well the chances rank the cases, not which threshold to use, and it doesn't change if every chance is halved.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks the AUC is the accuracy at some threshold, rather than how well the chances rank the real cases above the others.
- Thinks a forecast that can't tell the cases apart has an AUC of 0, when it has 0.5, the diagonal.
- Thinks each point on a ROC curve is one case, rather than one threshold's false positive rate and true positive rate.
- Thinks the AUC changes with the threshold you choose to use, when it sums up every threshold at once.
- Thinks changing the chances without changing their order, such as halving them all, changes the AUC.