Linear Algebra Essentials

In February 2023 Andrej Karpathy reported the most dramatic speedup he had found for nanoGPT, his compact GPT-2 training code: about 25%, from changing one number. The vocabulary size went from 50,257 to 50,304, the nearest multiple of 64. The model gained 47 rows of parameters that no token ever uses, and every training step got faster, because the matrix multiplication that produces the logits, in his words, "goes down a different kernel path with much higher occupancy": the GPU runs a different routine for it, one that keeps far more of its arithmetic units busy.