Language models and perplexity
Not met yet
A language model gives the chances of each possible next token. A bigram model looks one token back: count each pair, add a little smoothing to every count so nothing is impossible, and divide each row by its own total. Score it on text it never saw with the average of minus the log of the chance it gave each real next token; perplexity is e to that, and it reads as how many tokens the model is choosing between. Guessing evenly scores the vocabulary size, and lower is better. To write, start from a boundary token and draw each next token from the row of the one before, with a seeded generator so the same seed gives the same text.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks dividing counts by counts.sum(axis=0) makes each row of chances add up to 1.
- Thinks a pair never seen in the training text gets a small chance automatically, without smoothing.
- Thinks a next character the model gave no chance at all adds nothing to the loss, rather than an infinite amount.
- Thinks each sampled character should be drawn from the start character's row, rather than from the row of the character just drawn.
- Thinks a higher perplexity means a better language model.