Paul's Python CoursePPC

Starting weights

Not met yet

A network's starting weights decide whether it can learn at all. Hidden neurons that start identical get identical gradients and stay identical forever, so all-zero or all-equal starts waste every neuron but one. Starts that are too big push sums into the flat parts of the sigmoid, where almost no blame gets back. Small random weights, with a spread of np.sqrt(2 / n_in) for a ReLU layer and np.sqrt(1 / n_in) for a sigmoid one, keep each layer's numbers about the same size; biases can start at 0.

Try it

main.py

Output

Press Run and what your program prints shows up here.

Easy to get wrong

  • Thinks hidden neurons that start with identical weights will drift apart during training, when they get identical gradients and stay identical.
  • Thinks weights that all start at the same value cannot change, when they change together and stay equal.
  • Thinks big starting weights give a network a head start, when they saturate its sigmoids and shrink its gradients.
  • Thinks starting every weight at 0 is a neutral, safe choice for a network with hidden layers.

Where it comes up