Context Windows

Llama 3.1 8B accepts a 131,072-token prompt, sixteen times the 8,192 tokens of Llama 3, and that 128K window was the headline of Meta's release. Serve the model on one H100 and you can indeed send it such a prompt. But the same GPU that holds 52 conversations of 8,192 tokens holds only 3 conversations of 131,072, a single long prompt takes at least 6.7 seconds to process before the first output token appears, and NVIDIA's RULER benchmark scores the model at 77.0 at that length, below the 85.6 that Llama 2 7B managed at 4,096 tokens. By RULER's definition, the model's effective context is 32K.