Section 1

Representation and Attention Foundations

Before Llama 3.1 8B computes anything, text becomes token IDs, each ID selects a row of a 128,256 × 4,096 embedding table, rotary position embeddings rotate the queries and keys, and 32 query heads attend through 8 shared key/value heads. This section works each of those steps at the published shapes and prices it in tokens per character, parameters, FLOPs and KV-cache bytes, the quantities every later chapter budgets.