Foundation Models

The same 1,000 questions, each of the form Is 17077 a prime number? Think step by step and then answer "[Yes]" or "[No]"., with exactly half of the numbers prime, scored 84.0% on one version of GPT-4 and 51.1% on the next. The first version worked through each number as asked. The second answered almost every question with a bare "No" and none of the step-by-step work it had been asked for. Three researchers at Stanford and UC Berkeley ran this test in 2023 on two dated snapshots, fixed versions named after their release dates: gpt-4-0314 from March and gpt-4-0613 from June. Any application calling the plain model name gpt-4 had been moved from the first to the second on 27 June 2023, when OpenAI pointed that name at the new snapshot. Not one line of the application's code had changed.