Statistics

Llama 3.1 405B's 89.0% on HumanEval carries an error bar of ±4.8 points, wide enough that a rival scoring 91.5% on the same 164 problems would sit comfortably inside it. The formula is one line, 1.96√(S(1-S)/N) for a score S on a benchmark of N questions, and Meta's July 2024 paper on the Llama 3 family did something few model releases do: it printed intervals from that formula next to its benchmark scores, and noted that they understate the true uncertainty, because the choice of questions is not the only source of variation.