Paul's Python CoursePPC

Softmax and cross-entropy

Not met yet

A network that chooses one answer from many ends with a score per class and a softmax: shift each row by its own biggest score, exponentiate, and divide by the row's total, so each row becomes chances that add up to 1. Its loss is cross-entropy: minus the log of the chance given to the right answer, averaged over the rows, which is 2.303 for a brain guessing among ten. Together they have a tidy gradient: at the scores, the chances minus the one-hot answers, over the number of rows. The answer is the biggest column, argmax(axis=1).

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks exponentiating big scores straight away is safe, when np.exp overflows to infinity and the chances come out as nan.
  • Thinks softmax should make each column add up to 1, rather than each row, which holds one example's chances.
  • Thinks subtracting the biggest score in the whole batch protects every row, when a row of much smaller scores can be shifted until every exp is 0.
  • Thinks cross-entropy adds up the logs of every class's chance, rather than using only the chance given to the right answer.
  • Thinks log loss charges a confident wrong answer about the same as a hesitant one, when it grows without limit as the chance given to what happened falls to 0.
  • Thinks argmax(axis=0) picks each row's biggest column, when it picks each column's biggest row.

Where it comes up