Neural Networks

Llama 3.1 8B, the model Meta released in July 2024, is a list of 8,030,261,248 numbers. Count where they sit and a surprise appears: 5,637,144,576 of them, 70.2%, belong to 32 feed-forward blocks, and each block does little more than multiply its input by weight matrices and bend the results with a nonlinear function. LeNet-5 read handwritten cheque digits in 1998 with the same recipe and 60,000 numbers. By the end of this chapter you will have trained a 2,410-number version of it in numpy on 1,797 real handwritten digits.