The Chain Rule

The GPT-3 paper reports that training its 175-billion-parameter model on 300 billion tokens took 3.14 × 10²³ floating-point operations. Multiply the two counts and then by 6: 6 × 175 × 10⁹ × 300 × 10⁹ = 3.15 × 10²³. The rule behind that 6 is simple. Every weight costs 2 operations per token in the forward pass, a multiply and an add, and 4 more in the backward pass, which computes the gradient. The backward pass costs twice the forward pass because of the way the chain rule is evaluated, and that factor sets the price of training every model in this book.