The Gradient

Every optimizer step in a GPT-2 small training run ends with a little arithmetic on one vector. The vector has 124,439,808 entries, one for every weight in the model. The training script measures its length and, if the length exceeds 1.0, shrinks it. Andrej Karpathy's build-nanogpt, a 2024 from-scratch reproduction of GPT-2 small, does this with one call, torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0). The GPT-3 paper clipped at the same threshold in 2020, and Meta's Llama 2 and Ai2's OLMo 2 still did in 2023 and 2024. The vector is the gradient of the loss, and a training run changes a model by following it.