Few-shot Prompting

Put four labelled movie-review sentences in a prompt and ask GPT-3 2.7B to classify a fifth as positive or negative. Change nothing but the order of the four, and accuracy on the SST-2 sentiment benchmark lands anywhere from 54.3%, barely better than a coin, to 93.4%, close to the best systems of the time. Tony Zhao and colleagues measured that range in 2021. Working at the same time, Yao Lu and colleagues found the same order sensitivity at every model size they tested, not caused by any particular subset of examples, and an order that worked well for one model did not transfer to another.