Attention Mechanisms

In 2014 a neural translation system read an English sentence into one fixed-length vector and produced the French translation from that vector alone. Short sentences survived the squeeze. On long ones, performance "dramatically drops", in the words of Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, who measured the damage on the WMT'14 English-to-French task. Their encoder–decoder baseline, whose summary vector held 1,000 numbers, scored 17.82 BLEU (a 0–100 measure of n-gram overlap with reference translations). A model with the same hidden sizes and training data scored 26.75 once the decoder could look back at every source word each time it produced a target word and decide, with a learned score, how much each one mattered. They called the weights an alignment. Everyone now calls the mechanism attention.