Trading Voice Quality for Reliability

One of our customers runs a recruitment product where candidates talk to an AI by voice. The whole pitch is the conversation. It should feel like talking to a person, not navigating a phone tree. And users were getting frustrated and dropping out. The voice sounded natural, but the agent kept failing to do things mid-conversation: pull up the right information or trigger the right action on screen. A natural-sounding agent that doesn't do what you asked is worse than a clunky one that does.

The founder was worried, and he was right to be. For a product whose entire value is trust in a conversation, this was the one thing that couldn't be broken.

Tracing it to the architecture

We had built on OpenAI's Realtime API: speech in, speech out, with one model handling the conversation and tool calls. The demos were strong, but the user complaints clustered around actions that did not fire reliably. The transcript could read naturally while the requested change on screen failed. Documentation and community reports from that period described similar problems with tool calling on the Realtime preview.

We first tuned the prompt and rewrote the tool descriptions. Reliability improved a little, then plateaued. More prompt work was no longer producing evidence that the architecture could meet the product requirement.

Architecture tradeoffThe product was not buying voice quality in isolation

CaseWhat it winsWhat it costsProduction risk

Speech-to-speech realtimeLowest-feeling latency, natural turn shape, best demo.Harder to inspect and steer every intermediate decision.When tool calls misfire, the failure hides inside one fused model session.

Stitched STT → LLM → TTSInspectable transcript, mature text tool calling, replaceable stages.More moving parts and a noticeable beat of latency.Latency and orchestration become your responsibility.

Reframing it as a product decision

We reframed the choice with the founder around what candidates came to do. They could tolerate a short pause, but a missed action broke the conversation and sometimes ended it. That made tool reliability the primary requirement and voice naturalness a secondary one.

We researched Pipecat and LiveKit and spent about a month building a stitched-pipeline POC: speech-to-text, a text LLM with tool calling, and text-to-speech. It added latency and gave us three components to operate instead of one.

We built a test set of statements a candidate might say and measured tool-trigger reliability on the pipeline. Around 40 of 50 statements triggered the right tool in our first baseline pass. That result came from our rough internal harness rather than a published benchmark. The misses were more informative than the score: most traced to overlapping tool descriptions that we could separate. We had not found an equally actionable pattern in the Realtime failures.

The cost was real: responses landed at roughly 1.2 seconds, noticeably slower than the Realtime path. We took the tradeoff with our eyes open, because for this workflow a beat of silence was survivable and a wrong action was not.

The important detail is that the 1.2 seconds became inspectable. In the fused path, "it felt slow" and "it chose the wrong action" were hard to separate. In the stitched path, each stage had a receipt. If STT was slow, we saw it. If the LLM spent the budget choosing a tool, we saw it. If TTS blocked playback, we saw it. That is the hidden value of an uglier architecture: it gives you places to put instruments.

Evidence ledgerWhat the decision rested on

Around 40 of 50 statements triggered the right tool in the first stitched baselinemeasured

Measured in our rough internal harness, not a public benchmark.

Responses landed at roughly 1.2 secondsmeasured

Observed in the POC path and judged against the workflow tolerance.

Realtime voice models have improved since thenbelieved

The category has moved quickly; I would rerun the same evals before making the same call today.

Failure bucketsWhy the first 10 misses were useful

CaseWhat the eval showedWhat became actionable

Overlapping toolsTwo tools described the same user intent with slightly different names.Collapse or separate tool boundaries, then add negative examples.

Missing screen stateThe model chose a valid action for the sentence but not for the current UI state.Pass the visible state into the decision turn and validate transitions in code.

Transcript ambiguityThe statement was short enough that the right action depended on prior context.Keep the prior turn and candidate record in the eval case, not only the latest utterance.

Source trailWhat this post is in conversation with

• LiveKit model architecturesOfficial distinction between an STT–LLM–TTS pipeline and direct speech-to-speech models. The guide retains the historical internal results without assigning latency to individual stages.
• Application-side function executionThe application executes a proposed function and returns its result; a model tool call is not the side effect itself.
• OpenAI Realtime and audio guideThe current official framing for realtime sessions, voice-agent sessions, tools, and conversation state.
• LiveKit turn detection documentationBreaks voice timing into VAD, endpointing, interruption handling, and speech scheduling; those are the variables behind the delivery failures described here.
• Pipecat OpenAI Realtime function calling issueA concrete example of realtime voice tool calling getting tangled with context, transcription, and continuation behavior.
• LiveKit AgentsShows the current production components of a voice agent: STT/LLM/TTS pipelines, turn detection, tools, and tests.
• Need help building a real-time voice AI agentPractitioners compare platform choices through turn-taking and latency problems they encountered. I treat it as field context, not benchmark evidence.

What happened after

We shipped it. The reliability complaints stopped. And that "temporary" stitched architecture stayed the core of the product for about a year and a half, long past the point where anyone called it a compromise.

Continuing to patch prompts would have delayed an architectural change that the production evidence already supported. The 1.2-second response time was acceptable only because candidates valued a correct action more than the missing fraction of a second; another voice product could make the opposite trade.

Realtime voice models have improved since this decision. I would rerun the same evaluation before choosing an architecture today and let the new results, rather than the previous choice, decide.