Section 3

Scaling and Serving in Production

A trained Llama 3.1 becomes a product only when its context, number format, size and scheduler fit a latency target and a budget. This section computes those limits on H100s and smaller GPUs, from the live-token budget of a KV cache and the memory FP8 and INT4 save, through the cost of distilling a 70B teacher into an 8B student, to continuous batching, paged memory and speculative decoding, and closes with a review that sizes and prices one complete fleet.