Backpropagation

GPT-2's largest model has 1.5 billion parameters, and its weights take 3 GB in 16-bit precision. Training it with the Adam optimizer in mixed precision needs at least 24 GB before a single activation is stored, as Microsoft's ZeRO paper pointed out in 2019. Apply the same arithmetic to Llama 3.1 8B: its 16.1 GB of weights fit on one 24 GB card, but training it needs 128.5 GB before any activations, more than an 80 GB H100 holds. The factor of 8 comes from one algorithm and the machinery built around it: a gradient for every weight, and an optimizer that keeps three more numbers per weight in 32-bit precision.