Pretraining

Meta's report on Llama 3 describes its flagship training run in one sentence: a model with 405 billion trainable parameters, pretrained on 15.6 trillion tokens with 3.8 × 10²⁵ floating-point operations, on up to 16,384 H100 GPUs. You can check the third number from the first two with a rule that fits on a sticky note, six FLOPs per parameter per token: 6 × 405.85 × 10⁹ × 15.6 × 10¹² = 3.80 × 10²⁵. The model card adds the bill: 30.84 million H100-hours for the 405B model, 39.3 million for the family.