Multi-Head Attention

In 2019 Noam Shazeer timed a transformer translating English to German on a TPUv2. Generating output one token at a time for a batch of 1,024 sentences, the decoder cost 46 microseconds per generated token, and Shazeer traced the cost to memory bandwidth: every step reloaded the keys and values of all eight attention heads. He let the eight query heads share a single set of keys and values. Decoding fell to 3.8 microseconds per token, twelve times faster, and translation quality on the development set fell from 26.7 to 26.5 BLEU.