PyTorch
Not met yet
PyTorch is the brain you built upstairs, made big and fast. Tensors are arrays that can remember how they were made (requires_grad=True), and loss.backward() fills in every .grad, adding to what's there. A model is an nn.Module: layers in __init__, the route in forward, called as model(X); nn.Linear keeps its weight as (outputs, inputs). Every training step is zero, forward, loss, backward, step. nn.CrossEntropyLoss takes raw scores and does its own softmax, and a trained model is asked questions under torch.no_grad().
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks ending a PyTorch model with a softmax before nn.CrossEntropyLoss does no harm, when the loss softmaxes again and can never get near 0.
- Thinks nn.CrossEntropyLoss should be given chances that add up to 1, rather than the raw scores.
- Thinks optimiser.zero_grad() is optional tidying, when without it every step's gradient includes all the earlier ones.
- Thinks nn.Linear(64, 32).weight is shaped (64, 32) like the NumPy brain's W1, when PyTorch keeps it as (32, 64).
- Thinks optimiser.step() works out the gradients itself, so loss.backward() can be left out.