Evaluation platforms tend to grow into one script that loads cases, calls the model, scores the answers and decides whether to ship. The course's design pulls those jobs apart: separate case management, execution, grading, decision policy and evidence storage, so each can change without erasing lineage.
Five stages
- Version inputs. Bind the dataset, labels, slices, candidate artifacts and policies together.
- Run in isolation. Control seeds, retries, quotas, tool behavior and execution metadata.
- Grade with layers. Combine deterministic checks, calibrated model graders and targeted human review.
- Apply release policy. Evaluate hard gates, slice floors, uncertainty, latency and cost.
- Store and promote. Persist immutable evidence, and expose only approved candidates to shadow or canary traffic.
The figure's own caption states the rule that holds it together: the policy gate consumes evidence, and graders do not grant production authority directly.
Why the split matters
A grader produces a judgment, a rationale and some uncertainty. Whether that is good enough depends on things a grader does not know: which slices have floors, which gates are hard, what latency and cost the release can afford. Those belong to the release policy, which reads the stored evidence.
Mental Model
A grader is a witness and the release policy is the judge. Witnesses can be wrong, so their testimony is evidence with uncertainty attached, not a verdict.
Operating it
The course adds practical limits. Use asynchronous execution for expensive suites and deterministic job identities for deduplication. Isolate untrusted model or tool outputs. Enforce token, time, concurrency and monetary budgets per run. Store raw sensitive content only where policy permits, and otherwise keep references, redacted previews, hashes and aggregate results.
The evidence report should be reproducible from the manifest, or explicitly say which provider behavior cannot be replayed.
