Quantization

At 16 bits per parameter, the weights of Llama 3.1 405B alone occupy 811.7 GB, and the largest single machine most teams can rent, eight H100 GPUs with 80 GB each, holds 640 GB. When Meta released the model in July 2024, serving it in BF16 therefore meant splitting it across two machines and paying for the network between them on every generated token. Meta shipped a second checkpoint alongside the first: the same model with its feed-forward weights stored in 8-bit floating point. That version needs about 487 GB and fits on one machine.