Partial Derivatives

GPT-2 small ends every forward pass with a softmax over its 50,257-token vocabulary: 50,257 scores in, 50,257 probabilities out. Every probability depends on every score, so the derivative of that one step is not a single number but a table of 50,257 × 50,257 = 2,525,766,049 partial derivatives. In float32 that is 10.1 GB for one position of one sequence; a 1,024-token sequence would need over 10 TB. No framework ever builds that table. The backward pass needs only the table's product with one vector, and that product costs about 200,000 floating-point operations.