Calculus Essentials

GPT-2 small has 124,439,808 parameters. That count follows from its published configuration: 12 transformer layers, a width of 768, a vocabulary of 50,257 tokens and 1,024 positions. Every training step asks the same question of each of those numbers: if this one moved up slightly, would the loss go up or down, and how fast? The answer for one parameter is a derivative, and training needs all 124 million of them at every step.