Matrix Multiplication

When Llama 3.1 8B generates one token for one user, it multiplies a vector by nearly every weight matrix in the model: about 16 billion floating-point operations, two for each of its 8.03 billion parameters. An NVIDIA H100 SXM does 989 trillion of those per second in BF16, the 16-bit format models are usually served in, so the arithmetic takes about 16 microseconds. But those weights have to be read from memory first, roughly 16 GB, and at 3.35 TB/s that takes about 4.8 milliseconds. For 99.7% of each step the GPU's arithmetic units are waiting for data.