RLHF

People preferred the answers of a 1.3-billion-parameter model to those of the 175-billion-parameter GPT-3, a model more than a hundred times larger. OpenAI reported the result in March 2022 for InstructGPT, built from the 1.3B GPT-3 model. What it had in addition was about 13,000 demonstrations, rankings of answers to about 33,000 prompts written by roughly 40 contractors, and a reinforcement-learning stage that pushed the model toward answers a learned reward model scored highly. That recipe, reinforcement learning from human feedback, turned base models into assistants and is still the core of post-training.