Loss Functions

> Pair a sigmoid output with cross-entropy loss and a confidently wrong prediction produces a gradient of -0.99 -- a strong correction. Pair that same sigmoid output with mean squared error instead, and the identical mistake produces a gradient of only -0.0196 -- a 50-fold weaker correction, for the exact same error.