Batch normalisation
Not met yet
Batch normalisation puts every hidden neuron back on the same scale between layers. In training, for each column of Z (one neuron), it subtracts the batch's mean and divides by np.sqrt(var + eps), then multiplies by gamma and adds beta, two arrays the network learns. It also keeps running averages of the means and variances, momentum * old + (1 - momentum) * new. At evaluation it uses those running averages instead, because a batch of one evening is its own average and would come out as all zeros.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks batch normalisation leaves a batch of one row unchanged, rather than turning it into all zeros.
- Thinks a column with no spread makes batch normalisation give nan, forgetting the eps added to the variance.
- Thinks batch normalisation normalises across each example's neurons (rows) rather than down each neuron's column.
- Thinks batch normalisation uses the current batch's mean and variance at evaluation too, rather than the running averages.