Numbers, Functions & Graphs

Hugging Face's guide to evaluating language models runs GPT-2 large over the test split of WikiText-2, a benchmark built from Wikipedia articles, and reports a perplexity of 16.44. That one number summarizes the probability the model gave to every token of the test text (a token is a word or a piece of one), and the code that produces it never multiplies those probabilities together. It adds their logarithms, divides by the number of tokens and exponentiates at the end.