Agent Evaluation

The τ-bench repository includes recordings of Claude 3.5 Sonnet (the October 2024 release) handling each of the benchmark's 115 retail customer-service tasks eight times. On any single attempt it resolved 69.2% of them, a respectable score that would look fine on a slide. Now ask a different question of the same 920 conversations: on how many tasks did it succeed all eight times? The answer is 35.7%. For 67 of the 115 tasks, the agent sometimes got it right and sometimes did not, and a customer who brings that kind of request has no way to know which agent they will get.