Random Variables

Give a code model 100 attempts at a programming problem and count the problem solved if any attempt passes its unit tests. If each attempt had a 28.8% chance of passing, 100 independent attempts would all fail with probability 0.712¹⁰⁰ ≈ 1.8 × 10⁻¹⁵, and the model would solve essentially every problem. OpenAI's Codex-12B solved fewer than three quarters. The numbers come from HumanEval, the benchmark of 164 hand-written Python problems checked by unit tests that OpenAI introduced with Codex in July 2021: the 12-billion-parameter model solved 28.8% of the problems with one sample per problem, and 72.3% when it drew 100 samples and got credit whenever any of them passed.