LSTM Networks

Chapter 6 ended with a recurrent network that could not bridge 20 blank steps: shown a handwritten digit and asked to name it 20 steps later, it was right 15.6% of the time. Replace its tanh unit with a long short-term memory (LSTM) cell of the same width, train it with exactly the same recipe, and it is right 77.3% of the time. Train a second LSTM that differs only in one number at initialisation, the bias of its forget gate, and it is right 10.0% of the time: worse than the plain network it was meant to replace.