Softmax and cross-entropy
Not met yet
A network that chooses one answer from many ends with a score per class and a softmax: shift each row by its own biggest score, exponentiate, and divide by the row's total, so each row becomes chances that add up to 1. Its loss is cross-entropy: minus the log of the chance given to the right answer, averaged over the rows, which is 2.303 for a brain guessing among ten. Together they have a tidy gradient: at the scores, the chances minus the one-hot answers, over the number of rows. The answer is the biggest column, argmax(axis=1).
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks exponentiating big scores straight away is safe, when np.exp overflows to infinity and the chances come out as nan.
- Thinks softmax should make each column add up to 1, rather than each row, which holds one example's chances.
- Thinks subtracting the biggest score in the whole batch protects every row, when a row of much smaller scores can be shifted until every exp is 0.
- Thinks cross-entropy adds up the logs of every class's chance, rather than using only the chance given to the right answer.
- Thinks log loss charges a confident wrong answer about the same as a hesitant one, when it grows without limit as the chance given to what happened falls to 0.
- Thinks argmax(axis=0) picks each row's biggest column, when it picks each column's biggest row.