People ask where our eval test cases come from, expecting a methodology or a framework with a name. In our suites, most golden cases began as production failures.
We added one after an agent agreed to a rate it should have rejected. Another preserves the phrasing that caused a tool call to misfire. A third checks the two similarly named cities that an extraction once swapped. Each case keeps the input shape that exposed the failure.
The suite records our mistakes because production supplies cases our design reviews did not anticipate.
Why invented cases underperform
Cases written before launch reflect what the team already understands about the problem. Production adds unfamiliar phrasing, partial context, stale data, and interactions between components that were tested separately. Those failures expose gaps in the team's model of the workflow.
Our rule is to keep the evidence from each diagnosed failure. Production incidents, user complaints, and reviewer corrections are labeled examples acquired at real cost. Turning them into executable cases prevents the same input shape from surprising us again.
Case lifecycle
A fix is unfinished until the failure can be replayed
Follow the three failure classes named in the article. The diagram shows what evidence to keep without inventing private incident records.
Scenario
Article example; the original thread and private incident data are not reproduced here.
Sequence
- Keep the real failureCurrent
- Find the failing layerUpcoming
- Redact; preserve the shapeUpcoming
- Write the smallest checkUpcoming
- Ship fix and case togetherUpcoming
- Rerun on future changesUpcoming
Keep the real failure
The article describes an agent agreeing to a rate it should not have accepted.
Read the full explanation
Rate agreement
Article example; the original thread and private incident data are not reproduced here.
- Keep the real failure. The article describes an agent agreeing to a rate it should not have accepted.
- Find the failing layer. Inspect the proposal, applicable rate rule, and final action to identify the failing layer; do not infer the root cause from the reply alone.
- Redact; preserve the shape. Preserve the thread shape, relevant policy context, and action arguments while removing names, customer details, and identifiers.
- Write the smallest check. Assert that the resulting product action respects the applicable rate boundary or takes the required refusal/escalation path.
- Ship fix and case together. Keep the diagnosed fix, regression case, and a note explaining the failure in the same change. Preserve the case across architecture rewrites.
- Rerun on future changes. Replay the case against the fix and future candidate systems. A failure returns to diagnosis; a passing case stays in the suite.
Tool phrasing
Article example; this guide does not invent the missing private tool trace.
- Keep the real failure. The article describes a tool call misfiring on a particular phrasing.
- Find the failing layer. Compare the original phrasing with the selected tool and its arguments to establish what went wrong.
- Redact; preserve the shape. Preserve the triggering phrasing and tool arguments, redacting private data without rewriting the input into an easier case.
- Write the smallest check. Assert the expected tool selection and arguments for that preserved input. Prefer deterministic checks for these fields.
- Ship fix and case together. Keep the diagnosed fix, regression case, and a note explaining the failure in the same change. Preserve the case across architecture rewrites.
- Rerun on future changes. Replay the case against the fix and future candidate systems. A failure returns to diagnosis; a passing case stays in the suite.
Swapped cities
Article example; no city names or incident dates are fabricated.
- Keep the real failure. The article describes an extraction that silently swapped two cities with similar names.
- Find the failing layer. Inspect the source and extracted fields to locate where the two city identities were confused.
- Redact; preserve the shape. Preserve the confusing city-name relationship and relevant context while removing identifying customer material.
- Write the smallest check. Assert the expected origin and destination mapping against the preserved example, so a model swap cannot silently reverse them.
- Ship fix and case together. Keep the diagnosed fix, regression case, and a note explaining the failure in the same change. Preserve the case across architecture rewrites.
- Rerun on future changes. Replay the case against the fix and future candidate systems. A failure returns to diagnosis; a passing case stays in the suite.
Rate agreement
Article example; the original thread and private incident data are not reproduced here.
- Keep the real failure. The article describes an agent agreeing to a rate it should not have accepted.
- Find the failing layer. Inspect the proposal, applicable rate rule, and final action to identify the failing layer; do not infer the root cause from the reply alone.
- Redact; preserve the shape. Preserve the thread shape, relevant policy context, and action arguments while removing names, customer details, and identifiers.
- Write the smallest check. Assert that the resulting product action respects the applicable rate boundary or takes the required refusal/escalation path.
- Ship fix and case together. Keep the diagnosed fix, regression case, and a note explaining the failure in the same change. Preserve the case across architecture rewrites.
- Rerun on future changes. Replay the case against the fix and future candidate systems. A failure returns to diagnosis; a passing case stays in the suite.
Tool phrasing
Article example; this guide does not invent the missing private tool trace.
- Keep the real failure. The article describes a tool call misfiring on a particular phrasing.
- Find the failing layer. Compare the original phrasing with the selected tool and its arguments to establish what went wrong.
- Redact; preserve the shape. Preserve the triggering phrasing and tool arguments, redacting private data without rewriting the input into an easier case.
- Write the smallest check. Assert the expected tool selection and arguments for that preserved input. Prefer deterministic checks for these fields.
- Ship fix and case together. Keep the diagnosed fix, regression case, and a note explaining the failure in the same change. Preserve the case across architecture rewrites.
- Rerun on future changes. Replay the case against the fix and future candidate systems. A failure returns to diagnosis; a passing case stays in the suite.
Swapped cities
Article example; no city names or incident dates are fabricated.
- Keep the real failure. The article describes an extraction that silently swapped two cities with similar names.
- Find the failing layer. Inspect the source and extracted fields to locate where the two city identities were confused.
- Redact; preserve the shape. Preserve the confusing city-name relationship and relevant context while removing identifying customer material.
- Write the smallest check. Assert the expected origin and destination mapping against the preserved example, so a model swap cannot silently reverse them.
- Ship fix and case together. Keep the diagnosed fix, regression case, and a note explaining the failure in the same change. Preserve the case across architecture rewrites.
- Rerun on future changes. Replay the case against the fix and future candidate systems. A failure returns to diagnosis; a passing case stays in the suite.
Keep enough of the failure to replay it
Build a synthetic fixture for one of the article’s three failure classes. No private incident is reproduced; the symbolic assertion demonstrates the behavior the case must preserve.
tool == expected_tool && arguments == expected_arguments- Expected
- Choose the intended tool with the expected structured arguments.
- Candidate behavior
- The intended tool and arguments are selected.
Inspect the fixture’s provenance
The incident classes come from the article. Candidate behaviors and symbolic assertions are teaching fixtures, not reports of production test results. The rate fixture uses invented units and an inclusive 1800–2200 boundary: the original failure agrees above the ceiling, the regression agrees below the floor, and the fix rejects without agreeing. In a real case, retain the exact failing input, relevant retrieved context, tool arguments, and timestamps where needed; redact private details without removing the failure shape.
Attach a stable name, incident date, owner, and why the case exists. Prefer deterministic assertions for tool choice, canonical IDs, and policy boundaries; use a judge only when the behavior actually requires judgment.
The mechanics, such as they are
When we diagnose a failure, the fix includes the case that exposed it. We keep the actual input, anonymized where needed, and add an assertion for the behavior we expected. The code fix and the regression case belong in the same pull request.
Two details matter more than they look:
Keep the case after an architecture change. The old implementation may disappear while the underlying domain ambiguity remains.
Record why the case exists with the incident date and a one-line explanation. Unexplained cases become cleanup candidates once nobody remembers the production failure behind them.
One more detail I would add now: keep the trace shape, not only the text. If the failure involved retrieval, keep the retrieved context. If it involved a tool, keep the tool arguments. If it involved timing, keep the timestamps. A golden case that saves only the final user message is often a unit test for the wrong layer.
Name: short, stable, searchable. Prefer the failure class over the implementation detail.
Input: the original thread, message, transcript, document, or tool trace, redacted but not sanitized into something easier.
Expected: the product behavior that must survive model swaps: refuse, escalate, extract, update, or ask a clarifying question.
Assertions: deterministic checks first; judge rubric only for tone, usefulness, or other genuinely subjective output.
Note: incident date, owner, and the sentence explaining what pain this case prevents from coming back.
- OpenAI EvalsImplements evals as datasets paired with task-specific graders, including private cases for workflow-specific behavior.
- LangSmith EvaluationDocuments datasets built from curated cases, historical traces, or synthetic examples and the experiments run against them.
- PromptfooPractical CI-oriented reference for comparing prompts, models, providers, and red-team cases without treating examples as one-off anecdotes.
Over time, the suite becomes a history of failures that matter to this particular product. We run that history against each model migration and compare the new behavior with the contract captured in the cases.
We still add designed coverage and synthetic cases. The production cases carry something those methods cannot supply on their own: the exact context, phrasing, and component interaction that reached a user once.