Linear Regression

Fit ordinary least squares to scikit-learn's diabetes data and NumPy returns the answer in well under a millisecond: an intercept of −334.6, 5.60 units of disease progression per kg/m² of body-mass index, and a training mean squared error of 2,859.7. Now give the same problem to gradient descent, on the same raw columns, with the step size its theory recommends, one over the largest curvature of the loss. After 100,000 steps the training MSE is still 3,104, and the intercept has moved from its starting value of 0 to about −1 on its way to −334.6. Standardise the columns first and after 1,000 steps the same algorithm's training MSE is within 0.3 of the optimum. The data did not change. The shape of the surface the algorithm walks on did, and computing that shape is the first skill this chapter teaches.