Logistic regression
Not met yet
Logistic regression answers a yes-or-no question with a chance. It works out a straight-line score, z = X @ w + b, then squashes it with the sigmoid, 1 / (1 + np.exp(-z)), into a probability between 0 and 1: a score of 0 gives exactly 0.5, positive scores lean to yes and negative ones to no. It learns w and b by gradient descent on log loss, -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p)), which charges little for a hesitant miss and without limit for a confident one, and whose gradient is simply X.T @ (p - y) / len(y). The model only ever gives a chance; to make a decision, compare it with a threshold, usually 0.5.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks logistic regression answers 0 or 1 directly, rather than giving a probability that is then compared with a threshold.
- Thinks a score of 0 gives a probability of 0, when the sigmoid of 0 is 0.5.
- Thinks log loss charges a confident wrong answer about the same as a hesitant one, when it grows without limit as the chance given to what happened falls to 0.
- Thinks a straight line fitted to answers of 0 and 1 never gives a value below 0 or above 1.
- Thinks the miss in logistic regression's gradient is the score X @ w + b minus the answer, rather than the chance after the sigmoid minus the answer.
- Thinks the 0.5 threshold is compared with the score X @ w + b, rather than with the chance after the sigmoid.