Tokens, vocabularies and ids
Not met yet
A network takes numbers, so text is cut into tokens first: characters, words, or pieces of words. The vocabulary lists every different token once, often sorted(set(text)) for characters, and a token's id is its index in that list, counting from 0. Encoding turns text into ids with a dictionary from token to id; decoding turns ids back with the list. Characters make a small vocabulary that can spell any word, even a new one, at the cost of long sequences; words make short sequences, a huge vocabulary, and nothing to say about a word it has never seen.
Try it
main.pyOutput
Press Run and what your program prints shows up here.
Easy to get wrong
- Thinks sorted(set(text)) keeps the characters in the order they first appear.
- Thinks a space isn't a character, so it gets no place in the vocabulary.
- Thinks a vocabulary lists a character once for every time it appears.
- Thinks a token's id counts from 1, rather than being its index in the vocabulary from 0.
- Thinks a vocabulary of whole words can encode a word it has never seen.
- Thinks decoding characters should join them with spaces, rather than with nothing.