Paul's Python CoursePPC

Cross-entropy (log loss)

Not met yet

Cross-entropy, called log loss when the answer is yes or no, measures how wrong a chance was: for each case, minus the log of the chance the model gave to what actually happened, -(y * np.log(p) + (1 - y) * np.log(1 - p)), averaged over every case. Rain forecast at 0.9 costs 0.11; at 0.1 it costs 2.3; at 0 it costs without limit. Saying 0.5 every time costs log 2, about 0.69, which is the score any real model must beat. Paired with the sigmoid, its slope is simply the chance minus the answer, so a confidently wrong model gets a strong push, where mean squared error on chances barely nudges it. np.log(0) is minus infinity, so clip chances just inside 0 and 1 before taking logs.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks an evening without rain is charged minus the log of the chance of rain, rather than minus the log of the chance of no rain.
  • Thinks a forecast of 0.5 every time has a log loss of 0, when it costs log 2, about 0.69, whatever happens.
  • Thinks two models that make the same yes-or-no decisions must have the same log loss, whatever chances they gave.
  • Thinks mean squared error on the chances trains a classifier as well as log loss, when its slope all but vanishes for a model that is confidently wrong.
  • Thinks a chance of exactly 0 or 1 is safe in log loss, when np.log(0) is minus infinity and 0 times that is nan.

Where it comes up