Paul's Python CoursePPC

Activation functions

Not met yet

An activation function is applied to every neuron's weighted sum before it goes on. ReLU, np.maximum(0, z), bends: negatives become 0. Sigmoid, 1 / (1 + np.exp(-z)), squashes anything into 0 to 1, but past about 10 either way it is so flat that big differences vanish: it saturates. Without an activation between them, two layers collapse into one, and a network can only draw one straight line.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks ReLU turns negative numbers positive, rather than cutting them to 0.
  • Thinks sigmoid keeps big sums visibly apart, when past about 10 it gives practically 1 (or 0) for all of them.
  • Thinks a huge number inside np.exp stops the program with an error, when NumPy quietly gives 0 or infinity.
  • Thinks stacking layers with no activation between them lets a network draw curves one layer cannot.

Where it comes up