Across the client systems we run, two model providers are live in production: OpenAI's GPT series and Google's Gemini through the Vertex AI SDK. One call-analytics deployment handles transcription, summarization, sentiment, and intent on Vertex end to end; other systems mix providers by task. Supporting both affects prompts, evaluation, tracing, and incident response.
Why bother
The reasons are mostly operational.
Clients arrive with cloud commitments, and a GCP customer may require Vertex. Models also perform differently by task, with that difference changing between releases. A second provider gives us a migration path during sustained rate limits or degraded latency, although moving traffic still takes evaluation and prompt work. In practice, the advantage is movement measured in days instead of quarters, not instant failover.
Running two providers also forces the team to define an application contract: what the model receives, what it may return, which side owns retries, and what evidence each run leaves behind. That contract keeps provider-specific behavior at a boundary the team can inspect.
The myth of the portable prompt
I expected to reuse prompts with small provider-specific adjustments. In practice, the same instructions produce different formatting, refusal, and tool-calling behavior. One model may truncate where another embellishes; another may comply where the first refuses. We therefore version prompt variants by provider rather than treating the prompt as portable application code.
The eval suite transfers because its cases describe the product contract rather than a provider's behavior. We run the full suite before a provider or model change reaches a critical path. Aggregate improvements do not waive regressions on rate limits, tool arguments, or refusal behavior that matter to a particular workflow.
I would design the evaluation and tracing harness before the router. The provider adapter should record which model served a request, which prompt and tool contract it saw, what it tried to do, and how the result compared with the same case on the other path.
Provider comparison
Two adapters, one definition of correct
Follow the shared case through each prompt variant. Then introduce a missing field and inspect the common gate.
Scenario
Synthetic scheduling case and invented outputs. A and B are illustrative adapters, not results attributed to OpenAI or Gemini.
Sequence
- One product contractCurrent
- Adapter AUpcoming
- Adapter BUpcoming
- Same golden-case gateUpcoming
- Keep the evidenceUpcoming
One product contract
The same case asks to classify “schedule an interview.” It requires intent = schedule and a boolean needsFollowup. These are illustrative assertions, not a production schema.
Read the full explanation
Both satisfy the case
Synthetic scheduling case and invented outputs. A and B are illustrative adapters, not results attributed to OpenAI or Gemini.
- One product contract. The same case asks to classify “schedule an interview.” It requires intent = schedule and a boolean needsFollowup. These are illustrative assertions, not a production schema.
- Adapter A. Construct provider A’s request with its own prompt variant. Keep request, response, and error normalization narrow.
- Adapter B. Run the same case through a separate adapter and prompt. The branch is an evaluation comparison, not an automatic failover route.
- Same golden-case gate. Apply identical assertions to both normalized outputs. Change the selected case to see a missing field block one result while the product contract stays fixed.
- Keep the evidence. Keep provider/model identity, prompt and schema versions, raw payload, and the assertion result. A diagram cannot establish that a provider is ready for production traffic.
One field missing
Synthetic scheduling case and invented outputs. A and B are illustrative adapters, not results attributed to OpenAI or Gemini.
- One product contract. The same case asks to classify “schedule an interview.” It requires intent = schedule and a boolean needsFollowup. These are illustrative assertions, not a production schema.
- Adapter A. Construct provider A’s request with its own prompt variant. Keep request, response, and error normalization narrow.
- Adapter B. Run the same case through a separate adapter and prompt. The branch is an evaluation comparison, not an automatic failover route.
- Same golden-case gate. Apply identical assertions to both normalized outputs. Change the selected case to see a missing field block one result while the product contract stays fixed.
- Keep the evidence. Keep provider/model identity, prompt and schema versions, raw payload, and the assertion result. A diagram cannot establish that a provider is ready for production traffic.
Both satisfy the case
Synthetic scheduling case and invented outputs. A and B are illustrative adapters, not results attributed to OpenAI or Gemini.
- One product contract. The same case asks to classify “schedule an interview.” It requires intent = schedule and a boolean needsFollowup. These are illustrative assertions, not a production schema.
- Adapter A. Construct provider A’s request with its own prompt variant. Keep request, response, and error normalization narrow.
- Adapter B. Run the same case through a separate adapter and prompt. The branch is an evaluation comparison, not an automatic failover route.
- Same golden-case gate. Apply identical assertions to both normalized outputs. Change the selected case to see a missing field block one result while the product contract stays fixed.
- Keep the evidence. Keep provider/model identity, prompt and schema versions, raw payload, and the assertion result. A diagram cannot establish that a provider is ready for production traffic.
One field missing
Synthetic scheduling case and invented outputs. A and B are illustrative adapters, not results attributed to OpenAI or Gemini.
- One product contract. The same case asks to classify “schedule an interview.” It requires intent = schedule and a boolean needsFollowup. These are illustrative assertions, not a production schema.
- Adapter A. Construct provider A’s request with its own prompt variant. Keep request, response, and error normalization narrow.
- Adapter B. Run the same case through a separate adapter and prompt. The branch is an evaluation comparison, not an automatic failover route.
- Same golden-case gate. Apply identical assertions to both normalized outputs. Change the selected case to see a missing field block one result while the product contract stays fixed.
- Keep the evidence. Keep provider/model identity, prompt and schema versions, raw payload, and the assertion result. A diagram cannot establish that a provider is ready for production traffic.
Make the same case fail in a different way
Inject a drift into adapter B, change the gate, and inspect what reaches the application. Select any case to compare the full receipts.
Six invented recruitment cases, two fictional adapters. These results illustrate a test contract; they make no claim about any actual provider or model.
Read availability
Input: Show slots for candidate C-17; do not book.
A · reference receipt
{
"intent": "lookup",
"needsFollowup": false,
"tool": "list_slots",
"candidateId": "C-17"
}Accepted by the selected gate.
B · candidate receipt
{
"intent": "lookup",
"needsFollowup": false,
"tool": "book_slot",
"candidateId": "C-17"
}Blocked: tool: expected list_slots.
The same behavioral assertions catch the candidate regression. Preserve the failing input, output, and prompt version before changing the adapter.
Two smaller lessons
Keep the abstraction layer thin. We normalize requests, responses, and errors, and stop there. The heavyweight do-everything LLM-router frameworks average away each provider's strengths to give you a lowest-common-denominator interface. A thin shim plus per-provider prompt variants kept the strengths and cost us a little duplication, which is the right trade.
Log which provider and model version served every single request. The first time quality drifts and you can't answer "what changed?", you'll wish you had. Drift debugging without provenance is astrology.
Served by: provider, region, model name, model version if exposed, SDK version, and fallback path.
Prompt state: prompt version, tool schema version, retrieval index version, and output contract version.
Run shape: input tokens, output tokens, cached tokens if available, latency segments, retry count, and finish reason.
Debug handle: provider request ID, internal trace ID, tenant, workflow, and the eval case name if the run came from a gate.
The adapter is a quarantine layer
The adapter contains provider differences at one explicit boundary.
That means I want provider-specific code to be explicit and small: auth, endpoint shape, request construction, response parsing, retry classification, and usage accounting. Prompt variants live beside the provider, not buried inside a universal prompt builder. Tool-call parsing keeps the raw provider payload, even if the application gets a normalized object. Errors are normalized enough for operations, but the original error is still logged because "retryable provider error" is not enough when the provider changes behavior.
The mistake is building the abstraction too high. Once the adapter owns prompting, routing policy, eval interpretation, and fallback behavior, nobody can tell whether a regression came from the model, the prompt, the router, or the abstraction itself. Thin shims are less elegant. They are also easier to debug at 2 a.m.
- Promptfoo test case configurationPrimary documentation for cases with inputs and assertions. The diagram uses invented outputs to illustrate a shared gate, not measured provider behavior.
- Vertex AI release notesA reminder that provider behavior and model availability are moving targets, not static dependencies.
- Promptfoo Vertex AI providerDocuments how Promptfoo configures Vertex models inside the same test cases used for other providers.
- OpenAI function calling guideDefines the application-side sequence: receive a tool call, execute code, return the tool output, and continue the model turn.
Our experience covers model access through APIs and SDKs, not each provider's fine-tuning stack or vector platform. At that layer, the durable work is the harness: eval cases, logging, and application contracts. We can reuse those when a provider changes even though the prompts and adapters still need attention.