Self-Attention

Meta's model card gives every Llama 3.1 model, including the 8-billion-parameter one, a context window of 128K tokens. Take that at face value and hand Llama 3.1 8B a single 131,072-token prompt on one H100. Three very different bills arrive. Reading the prompt costs about 6.5 × 10¹⁵ floating-point operations, 70% of them in attention. Remembering the prompt while the model answers costs 16 GiB (17.2 GB) of keys and values, more than the model's 16.06 GB of weights. And if the GPU computed attention the way the formula is usually written, it would need 32 GiB for the score matrix of one head in one layer, a matrix that no production kernel ever stores.