GPT Evolution

In November 2019 OpenAI published the last of GPT-2's weights: 1.56 billion parameters in 48 identical decoder blocks, each normalised with LayerNorm, each running a GELU feed-forward layer, reading positions from a learned table that stops at 1,024 tokens. In August 2025 it published the weights of gpt-oss-120b. Put the two configurations side by side and almost every line differs. The new model normalises with RMSNorm, encodes position by rotating queries and keys, lets 64 query heads share 8 key-value heads, reads up to 131,072 tokens, and holds 116.8 billion parameters of which only 5.1 billion run for any given token, because a router sends each token to 4 of 128 expert networks.