One of our customers runs a freight platform where an AI agent negotiates loads over email with brokers. A bad negotiation can reduce margin and damage a broker relationship before anyone on our side sees the thread.
At first, we sampled threads by hand before shipping changes. That stopped scaling when traffic grew beyond what the team could read. The agent had become too eager to close and was agreeing to terms it should have rejected. We found the behavior late by reading affected threads. We could estimate the dollar impact from those conversations, but we never had a clean before-and-after measurement.
A pending model migration made the measurement gap urgent.
Counted in the eval suite we used for the migration gate.
Pulled from real negotiation threads and kept as named regression cases.
Estimated from affected conversations after the fact; we did not have a clean before/after measurement.
Production-volume counters across the tenants this system served.
The migration that forced the issue
We needed to move from GPT-4o to GPT-4.1. Aggregate model improvements did not tell us whether the new version would preserve our rate boundaries, tool calls, and negotiation stance. We also could not read ten thousand email threads to find regressions before customers did.
I pushed for an eval suite before the migration. It delayed roadmap work, but we had no repeatable way to compare the two models on negotiations that mattered to the business.
The suite ended up at 87 test functions and 19 golden cases pulled from real negotiations. The design decision I'd defend most is the split:
- Hard assertions for anything with a right answer. Rate boundaries, compliance rules, things the agent must never offer. No model judges these. Code does.
- LLM-as-judge only for tone. Is the reply professional? Does it read like a negotiator and not a chatbot? And before trusting the judge, we validated it against human ratings on the same samples. If the judge can't agree with a human, it doesn't get a vote.
I see teams do this the other way around: an LLM judge scoring everything, including the parts that have exact answers. That's how you end up with a flaky eval suite nobody trusts, which is worse than no suite, because it gives you confidence instead of information.
The judge also needs its own review loop. Treat it like a measuring instrument that drifts: collect human corrections, turn the sharp disagreements into examples, and track agreement over time. A judge prompt that sounds reasonable is not a release gate. A judge that has been calibrated against the cases your team actually argues about is closer to one.
Migration evidence
Good tone cannot overrule a failed invariant
Compare baseline and candidate traces through separate exact and qualitative checks, then inspect the release decision.
Scenario
Illustrative baseline and candidate traces. These are not the two historical regressions and do not reconstruct private test cases.
Sequence
- Compare frozen tracesCurrent
- Check exact invariantsUpcoming
- Review tone with calibrationUpcoming
- Apply the offline release gateUpcoming
- Keep runtime enforcement separateUpcoming
Compare frozen traces
Run the same representative input through the baseline and candidate. Keep proposed rates, tool arguments and replies beside each other. The illustrative baseline is within bounds with a valid load ID.
Read the full explanation
Polite policy failure
Illustrative baseline and candidate traces. These are not the two historical regressions and do not reconstruct private test cases.
- Compare frozen traces. Run the same representative input through the baseline and candidate. Keep proposed rates, tool arguments and replies beside each other. The illustrative baseline is within bounds with a valid load ID.
- Check exact invariants. The candidate proposes a below-floor offer while keeping a valid load ID. The rate assertion fails even though the reply is polite. Tool correctness does not cancel a policy violation.
- Review tone with calibration. The illustrative tone judgment is acceptable and the judge has been calibrated against human ratings. This qualitative result has no authority to override a failed rate check.
- Apply the offline release gate. Hold the candidate. A critical assertion failed; good tone is insufficient. This is an offline decision about deploying the candidate version.
- Keep runtime enforcement separate. For the same illustrative proposal, a server-side validator rejects the below-floor tool action at execution time. Offline evals catch regressions before release; runtime policy protects each action.
Uncalibrated judge
Illustrative baseline and candidate traces. These are not the two historical regressions and do not reconstruct private test cases.
- Compare frozen traces. Run the same representative input through the baseline and candidate. Keep proposed rates, tool arguments and replies beside each other. The illustrative baseline is within bounds with a valid load ID.
- Check exact invariants. The candidate’s rate stays within bounds and its tool arguments retain the required load ID. Exact checks run as code, without an LLM deciding whether arithmetic or policy is acceptable.
- Review tone with calibration. The judge calls the tone acceptable, but has not been validated against human ratings for this example. Treat that judgment as provisional; it cannot clear this release gate.
- Apply the offline release gate. Hold the candidate until the qualitative review is calibrated and disagreements are understood. Passing the exact checks alone does not complete the review.
- Keep runtime enforcement separate. For this illustrative proposal the runtime validator permits the in-bounds action with valid arguments. That local permission is distinct from approval to deploy a whole candidate version.
Cleared candidate
Illustrative baseline and candidate traces. These are not the two historical regressions and do not reconstruct private test cases.
- Compare frozen traces. Run the same representative input through the baseline and candidate. Keep proposed rates, tool arguments and replies beside each other. The illustrative baseline is within bounds with a valid load ID.
- Check exact invariants. The candidate’s rate stays within bounds and its tool arguments retain the required load ID. Exact checks run as code, without an LLM deciding whether arithmetic or policy is acceptable.
- Review tone with calibration. The illustrative tone judgment is acceptable and the judge has been calibrated against human ratings. This qualitative result has no authority to override a failed rate check.
- Apply the offline release gate. This simplified candidate clears its hard assertions and calibrated qualitative review. Real release decisions also inspect changed traces and retain known regression cases.
- Keep runtime enforcement separate. For this illustrative proposal the runtime validator permits the in-bounds action with valid arguments. That local permission is distinct from approval to deploy a whole candidate version.
Polite policy failure
Illustrative baseline and candidate traces. These are not the two historical regressions and do not reconstruct private test cases.
- Compare frozen traces. Run the same representative input through the baseline and candidate. Keep proposed rates, tool arguments and replies beside each other. The illustrative baseline is within bounds with a valid load ID.
- Check exact invariants. The candidate proposes a below-floor offer while keeping a valid load ID. The rate assertion fails even though the reply is polite. Tool correctness does not cancel a policy violation.
- Review tone with calibration. The illustrative tone judgment is acceptable and the judge has been calibrated against human ratings. This qualitative result has no authority to override a failed rate check.
- Apply the offline release gate. Hold the candidate. A critical assertion failed; good tone is insufficient. This is an offline decision about deploying the candidate version.
- Keep runtime enforcement separate. For the same illustrative proposal, a server-side validator rejects the below-floor tool action at execution time. Offline evals catch regressions before release; runtime policy protects each action.
Uncalibrated judge
Illustrative baseline and candidate traces. These are not the two historical regressions and do not reconstruct private test cases.
- Compare frozen traces. Run the same representative input through the baseline and candidate. Keep proposed rates, tool arguments and replies beside each other. The illustrative baseline is within bounds with a valid load ID.
- Check exact invariants. The candidate’s rate stays within bounds and its tool arguments retain the required load ID. Exact checks run as code, without an LLM deciding whether arithmetic or policy is acceptable.
- Review tone with calibration. The judge calls the tone acceptable, but has not been validated against human ratings for this example. Treat that judgment as provisional; it cannot clear this release gate.
- Apply the offline release gate. Hold the candidate until the qualitative review is calibrated and disagreements are understood. Passing the exact checks alone does not complete the review.
- Keep runtime enforcement separate. For this illustrative proposal the runtime validator permits the in-bounds action with valid arguments. That local permission is distinct from approval to deploy a whole candidate version.
Cleared candidate
Illustrative baseline and candidate traces. These are not the two historical regressions and do not reconstruct private test cases.
- Compare frozen traces. Run the same representative input through the baseline and candidate. Keep proposed rates, tool arguments and replies beside each other. The illustrative baseline is within bounds with a valid load ID.
- Check exact invariants. The candidate’s rate stays within bounds and its tool arguments retain the required load ID. Exact checks run as code, without an LLM deciding whether arithmetic or policy is acceptable.
- Review tone with calibration. The illustrative tone judgment is acceptable and the judge has been calibrated against human ratings. This qualitative result has no authority to override a failed rate check.
- Apply the offline release gate. This simplified candidate clears its hard assertions and calibrated qualitative review. Real release decisions also inspect changed traces and retain known regression cases.
- Keep runtime enforcement separate. For this illustrative proposal the runtime validator permits the in-bounds action with valid arguments. That local permission is distinct from approval to deploy a whole candidate version.
Try to pass a polite policy violation
Move the offer across a hard boundary, then test whether the tone judge agrees with the human labels it depends on.
Rate violates the server-side bounds.
A hard invariant failed.
Review all eight tone samples and change a human label
These short invented replies isolate tone. Rate and tool correctness are evaluated separately. Changing a human label updates the agreement matrix immediately; the selected judge’s predictions stay fixed. The more accepting judge has two false accepts only against the original human labels.
Thanks. We can offer the approved rate and confirm after your response.
Judge: acceptThat falls outside the approved range. I can ask for a review.
Judge: acceptObviously you should have accepted our first offer.
Judge: rejectCould you confirm the pickup window before we proceed?
Judge: acceptWhatever. Take it or leave it.
Judge: rejectJust say yes now. You are wasting our time.
Judge: rejectThank you for the counteroffer. Here is the next available step.
Judge: acceptWe do not care about your constraints.
Judge: reject
Illustrative gate and eight invented tone samples, not the article’s private suite or caught regressions. Observed agreement on a tiny sample does not establish general judge reliability; production calibration needs representative, held-out cases and disagreement review. Passing an offline sample and allowing one runtime action are separate decisions.
Get the risky logic out of the prompt
The evals exposed a second problem. Some negotiation behavior lived in the prompt, which means it lived at the model's discretion. Discretion is exactly what you don't want around money.
So we moved the negotiation policy (what can be offered, when, within what bounds) out of the prompt and into server-side code. The model drafts the language; the policy decides the numbers. After that change, a whole class of failure became impossible rather than unlikely. A prompt can drift with a model version. An if statement doesn't.
# The model never decides the rate. Policy does.
def validate_offer(offer: Offer, load: Load) -> Verdict:
if offer.rate < load.floor_rate:
return Verdict.reject("below floor")
if offer.rate > load.ceiling_rate:
return Verdict.reject("above ceiling")
return Verdict.allow()That snippet is illustrative rather than production code. It shows the boundary we used: the model drafts language while deterministic policy validates the rate.
Input: a broker thread where a counteroffer sounds acceptable in plain English but violates the load's rate floor.
Expected behavior: draft a polite refusal or escalation path; never emit a tool call that commits the below-floor rate.
Assertions: rate validator rejects the offer, tool arguments preserve the load ID, and the reply explains the next acceptable action without inventing a number.
Why it exists: this exact shape reached production once. The case stays even if the model, prompt, or tool layer changes.
- Evals for LLM FeaturesSeparates deterministic assertions from judge-scored qualities and uses human review to calibrate the latter.
- LLM-as-a-judge: calibrate before you trust the scoreArgues that a judge prompt needs agreement checks against human labels before it can gate a release.
- How to Calibrate LLM-as-Judge with Human CorrectionsDescribes collecting human corrections, adding them as examples, and tracking judge agreement over time.
- LLM Evaluation in ProductionCovers dataset versioning, baseline resets, and the different jobs of CI evals and production sampling.
Migration results
We ran the GPT-4o to 4.1 migration through the suite. It caught two real regressions that would otherwise have reached brokers. We fixed both before cutover, and customers did not see them.
That eval loop is what lets the system run at the scale it runs today, 80,000+ emails and 10,000+ loads across five production tenants, without anyone reading threads all day hoping to spot trouble.
Evals became part of the release mechanism rather than a check performed at the end. They gave model, prompt, and policy changes the same set of cases and made regressions visible before deployment.
Any model change that can affect money, safety, or a customer's workflow now runs against written invariants and production-derived cases before release.
If a model migration cannot be evaluated before cutover, the team is still depending on customers to identify changed behavior. Building the cases before the swap is cheaper than reconstructing them from an incident.