Derivatives

PyTorch keeps its calculus in one file. tools/autograd/derivatives.yaml has an entry for every operation the framework can differentiate, and the entries are short. Leaving out a .conj() that matters only for complex numbers, the backward rule for exp is grad * result. For log it is grad.div(self), for sqrt it is grad / (2 * result), and for sigmoid it is sigmoid_backward(grad, result). Each is a derivative you will derive in this chapter, multiplied by grad, the gradient arriving from later in the network.