Positional Encoding

When Meta released Llama 3.1 in July 2024, the 8B model had the same layers, the same tokenizer and the same weight shapes as Llama 3 8B, which had shipped three months earlier with an 8,192-token context window. Llama 3.1 advertised 131,072 tokens. Part of what made the difference is a twelve-line Python function in Meta's reference code, apply_scaling, which changes 35 of the 64 rotation frequencies in every attention head and leaves the rest alone. The other part is roughly 800 billion tokens of long-context training.