Dropout
Not met yet
Dropout switches off each hidden neuron with chance rate on every training step, choosing afresh each time, so no neuron can rely on any particular other one and the network can't build a single fragile path through the training data. A mask does it: keep = rng.random(H.shape) >= rate, then H * keep / (1 - rate). Dividing by 1 - rate scales the survivors up so the layer's total stays about the same on average. The backward pass uses the same mask. At evaluation nothing is dropped and nothing is scaled.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks dropout picks which neurons to switch off once, and the same ones sleep every step.
- Thinks the neurons left awake by dropout pass through unchanged, rather than divided by 1 - rate.
- Thinks dropout turns every neuron down by the same fraction, rather than switching some fully off.
- Thinks dropout should stay switched on when the network is evaluated or used.