Loss Functions

Start a Llama 3 model training from random weights and the first loss it prints should be close to 11.76. The number is not a coincidence or a property of Meta's data. Llama 3's tokenizer has 128,256 tokens, a model that has learned nothing spreads its probability roughly evenly across them, and the cross-entropy of an even guess over 128,256 options is ln 128,256 = 11.76. The digits network of Chapters 1 and 2 starts at ln 10 = 2.30 for the same reason. Andrej Karpathy's widely read training checklist includes this check: verify that the loss at initialisation equals -ln(1/n_(classes)). If yours prints 25, the output layer is wildly overconfident before it has seen a single example; if it prints 0.3, the labels are leaking into the input.