End of Volume Review & Interview Questions

About $20 buys a from-scratch reproduction of GPT-2 small: 90 minutes on one node of eight A100 GPUs, rented at $14 an hour, training on 10 billion tokens. Andrej Karpathy reported that cost in May 2024 for llm.c, his C and CUDA implementation of the training loop, and build-nanogpt, the PyTorch repository behind his video reproduction of the same model, publishes every setting. Read the settings with this volume in hand and almost nothing in them is arbitrary. The 124,439,808 parameters sit in the shapes Chapter 2 laid out. Attention divides its scores by √64 = 8 for the variance reason in Chapter 3. Each training token costs about 6N floating-point operations, the count from Chapters 5 and 11. An untrained model should report a loss near 11 nats, just above ln 50,257 = 10.82, the anchor from Chapters 10 and 18. AdamW with β₂ = 0.95 and a peak learning rate of 6 × 10⁻⁴ comes from Chapter 12's table. And the closing claim of Karpathy's write-up, "We get to 29.9 here, which surpasses GPT-2 (124M) at 29.4" on the HellaSwag benchmark, is a comparison of two proportions measured on the same 10,042 questions, which is Chapter 17's subject.