The intended lesson is simple: escalate when a customer mentions a legal threat. A team writes fifty training examples to teach it. All fifty come from the same reviewer, who happens to write escalations more formally.
ESCALATE "I will be contacting my solicitor."
-> "I am escalating this to our specialist team, who will
respond within one business day."
NON-ESCALATE "My order is late again!"
-> "Sorry about that! Let me check."
What the data actually teaches
A model learns whatever correlates with the label, including correlations nobody intended. In these fifty examples, four things predict "escalate", and only one of them was meant:
- legal language, the intended feature;
- message length, by accident, because the threats happened to be longer;
- the absence of an exclamation mark, by accident;
- a formal register in the target reply, by accident: it is the reviewer's own habit.
The model can reach 100% training accuracy using length and punctuation alone, and it will, because those are easier features than legal intent. In production, a long, formal, punctuation-free complaint about delivery windows gets escalated, and a terse "solicitor!" does not.
Hold the decision, vary everything else
For each escalation, add a near neighbor that shares its surface features and inverts the label: a long, formal message that needs no escalation, a terse legal threat, an escalation written casually. These pairs are what force the model onto the feature you meant, and they are the examples nobody writes, because they feel redundant.
Common Mistake
A training set built only from confident, complete answers teaches the model that every input deserves one. If the product depends on asking a clarifying question, abstaining, citing, calling a tool, routing to a person or refusing safely, each of those decisions needs examples of its own.
The same care applies to what a target is allowed to know. An example whose answer uses facts from a ticket record the model will not see at serving time teaches it to state those facts confidently, which means inventing them. Teach the tool call that fetches the facts instead, and teach abstention explicitly for the case where no tool or evidence is available.
