Precision, recall and the confusion matrix
Not met yet
A yes-or-no forecast is right in two ways and wrong in two: true positives (warned, and it came), false positives (a false alarm), false negatives (a miss) and true negatives. Those four counts are the confusion matrix. Precision starts from the warnings, tp / (tp + fp): the share that were right. Recall starts from the real cases, tp / (tp + fn): the share that were caught. F1 is their harmonic mean, high only when both are. When the thing you're after is rare, accuracy flatters: a forecast that never warns of a storm that comes 25 nights in 400 is still 94% accurate, and catches nothing. Raising the threshold warns less often, which tends to lift precision and can only lower recall.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks a model that ignores its readings and always gives the same answer scores about 50% accuracy, whatever the answers are.
- Thinks accuracy only counts the cases where the model says yes, so a model that never says yes scores nothing.
- Thinks precision is the share of real cases that were caught and recall the share of warnings that were right, the wrong way round.
- Thinks accuracy is a fair score even when one answer is far rarer than the other.
- Thinks F1 is the ordinary average of precision and recall, rather than their harmonic mean, which stays low if either is low.
- Thinks raising the threshold catches more of the real cases, when it means warning less often, so recall can only fall or stay the same.