Section 1

Neural Network Foundations

Section 1 builds a neural network from one layer and trains it in numpy on scikit-learn's 1,797 real handwritten digits. Chapter 1 shows why layers need a nonlinearity between them and counts real networks from LeNet-5 to Llama 3.1 8B, Chapter 2 chooses the nonlinearity by its derivative, Chapter 3 derives the loss and its gradient, and Chapter 4 carries that gradient backwards and prices a training run at 16 bytes per parameter. The numbers the chapters predict, from a starting loss of 2.30 on the digits and 11.76 for Llama 3 to 128.5 GB of training state for Llama 3.1 8B, are checked by programs you can run.