LangChain

Sierra's τ-bench hands an agent a written retail policy, sixteen tools over a database of 500 customers and 1,000 orders, and a simulated customer who wants something done. One line of that policy leaves no room for interpretation: before any action that changes the database, list the details and obtain an explicit yes. In the trajectories Sierra published for gpt-4o, the agent made 666 database-changing tool calls across 460 conversations, and 80 of them (12.0%) came straight after a customer message that contained no affirmative word at all: no "yes", no "sure", no "ok", no "go ahead". For Claude 3.5 Sonnet (October 2024) the count is 68 of 1,352, or 5.0%. The rule sat in the system prompt, and the model followed it most of the time.