Gradient Descent & Optimization

Eight H100 GPUs can train GPT-2 small to a validation loss of 3.28 on the FineWeb dataset in 1.126 minutes, using under 400 million training tokens. When Andrej Karpathy's llm.c reproduction first reached that loss in May 2024, it needed 10 billion tokens and 45 minutes on the same hardware (90 on the eight A100s of Chapter 11's estimate), and its run became the starting line of a public competition, the NanoGPT speedrun: same hardware, same target loss, fastest wins. Nine days later a retuned learning rate and rotary embeddings had cut the record to 31.4 minutes. The third record changed the optimizer: Muon, credited to Keller Jordan and Jeremy Bernstein, replaced AdamW in October 2024 and brought the time to 24.9 minutes. Dozens of changes produced the rest, down to the record of 6 August 2026 that stands as of September 2026, but the optimizer was among the first, and it is the part of training that the gradient alone does not decide.