Model Serving

One H100 serving Llama 3.1 8B can generate at most about 207 tokens per second for a single user in a short conversation: every token requires reading all 16 GB of weights, plus the conversation's cache, and the GPU reads 3.35 TB per second. Give the same GPU 185 chat conversations at once and, in the same second, it can generate about 9,000 tokens, while each user still sees a new token every 20 milliseconds, faster than anyone reads. Nothing about the model changed. The difference is how the requests share the weight reads, and that factor of about 44 is most of the economics of running language models.