Probability Essentials

When pre-trained GPT-4 gave an answer a 70% chance of being right, it was right about 70% of the time. After post-training with reinforcement learning from human feedback, the same model's stated probabilities drifted more than ten times further from its accuracy. OpenAI's technical report of March 2023 printed the two measurements side by side as small calibration plots. Both came from the model answering the same subset of MMLU multiple-choice questions: for every question the authors took the probability the model assigned to each answer letter, grouped the answers by that probability, and checked how often each group was right. The pre-trained model's points sat on the diagonal, with an expected calibration error of 0.007; the post-trained model's gave 0.074. The report's caption is blunt: post-training hurts calibration significantly.