Activation functions
Not met yet
An activation function is applied to every neuron's weighted sum before it goes on. ReLU, np.maximum(0, z), bends: negatives become 0. Sigmoid, 1 / (1 + np.exp(-z)), squashes anything into 0 to 1, but past about 10 either way it is so flat that big differences vanish: it saturates. Without an activation between them, two layers collapse into one, and a network can only draw one straight line.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks ReLU turns negative numbers positive, rather than cutting them to 0.
- Thinks sigmoid keeps big sums visibly apart, when past about 10 it gives practically 1 (or 0) for all of them.
- Thinks a huge number inside np.exp stops the program with an error, when NumPy quietly gives 0 or infinity.
- Thinks stacking layers with no activation between them lets a network draw curves one layer cannot.