Activation Functions

In November 2021, in the first days of full-scale training runs for the 175-billion-parameter OPT model on 1,024 A100 GPUs, Meta's team kept hitting the same wall: gradient norms exploded, the loss spiked, values turned into NaN, and the FP16 loss scale collapsed until it underflowed. Among the drastic measures they logged was swapping the activation function in the middle of the run, replacing GELU with ReLU, "noting that the x**3 term in GeLU might be a source of instability with FP16". Nine years earlier, AlexNet's authors had reported that a network with ReLUs converged about six times faster than an equivalent network with tanh units. And inside every feed-forward block of Llama 3.1 today sits a sigmoid, the function the field had abandoned for hidden layers, now doing a different job as a gate.