Eigenvalues & Eigenvectors

Multiply a vector by the same matrix a thousand times. If the matrix stretches some direction by 0.99, the vector's component along that direction ends at 0.99¹⁰⁰⁰ ≈ 4.3 × 10⁻⁵ of where it started. If it stretches by 1.01, the component ends 20,959 times larger. A 2% difference in one number separates a signal that disappears from one that explodes, and that is exactly what happened to gradients flowing backwards through early recurrent networks.