When a normal program fails, you get a stack trace pointing somewhere. When a voice agent fails, you get a complaint ("it ended the call on me", "it didn't book the load", "it asked for my number three times"), a transcript, a tool-call log, and a feeling of dread. The failure is in there somewhere, spread across four systems that each swear they did their job.
After working through these incidents across several voice deployments, I now inspect the same layers in the same order. The first investigations were slower because I followed the complaint rather than the event sequence.
Read the layers in order
A voice conversation passes through audio capture, speech-to-text, the model's decision, tool execution, text-to-speech, and playback. Users experience failures at any of those layers as one conversation going wrong. I read the evidence in this order:
First, the transcript against the audio intent. Did the system hear what was said? Misrecognized entities are the classic: a company name mangled by STT, a number transcribed with a digit dropped. If the transcript is wrong, stop. Nothing downstream had a chance, and no prompt edit will fix a hearing problem.
Second, the decision against the transcript. Given what the system heard, was the LLM's move reasonable? This is where you check the tool calls: did it call the right tool, with the right arguments, at the right turn? A model that heard correctly and chose wrongly is a genuine reasoning or prompting failure. This is the only layer where prompt work is even relevant, and in my experience it's the guilty layer less often than people assume.
Third, the tool result against the decision. The model called the right thing and the backend returned something stale, empty, or slow. The agent then improvised around a bad answer, which reads to the user as confusion. The model takes the blame for the API's sins constantly.
Fourth, the delivery. Right hearing, right decision, right data, and the turn still failed: the response landed too late, talked over the caller, or the call-end logic fired early. Timing failures are invisible in transcripts, which only record what was said, not when. You need the timestamps, and you need to actually look at them.
The fastest way to waste an afternoon is to debug from the complaint label. "It ended the call on me" sounds like call-end logic until you find the agent asked a confirmation question, the caller answered during TTS playback, the interruption was ignored, and the silence timer fired because the system thought the user had not responded. The symptom was call ending. The bug was turn-taking state.
Call forensics
Find the first layer that changed the intent
Follow an invented book/look trace across speech, transcript, decision, and playback. It applies the article’s method: read each layer against the evidence it actually received.
Scenario
Invented worked example applying the article’s debugging order. Utterances and timestamps are illustrative; spacing is schematic. No recorded audio is shown.
Sequence
- Establish intentCurrent
- First divergenceUpcoming
- Read downstream fairlyUpcoming
- Then inspect timingUpcoming
Establish intent
At 00:12.180 in this invented trace, the caller says “book the 10 a.m. slot.” Treat this as the example’s reference intent, not as a quote from a recorded call.
Read the full explanation
Book → look
Invented worked example applying the article’s debugging order. Utterances and timestamps are illustrative; spacing is schematic. No recorded audio is shown.
- Establish intent. At 00:12.180 in this invented trace, the caller says “book the 10 a.m. slot.” Treat this as the example’s reference intent, not as a quote from a recorded call.
- First divergence. At 00:12.910, “book” becomes “look,” with low confidence on the first token. This is the first evidence of intent changing between layers. Start the investigation at hearing.
- Read downstream fairly. At 00:13.240, the model asks for clarification rather than calling the booking tool. Assess that move against what it received: the already changed transcript. No booking tool execution is present in this example.
- Then inspect timing. At 00:14.800, this example assumes TTS starts while the caller is correcting the reply. Delivery adds a second failure. The correction’s start time and playback duration are omitted, so these events cannot establish an overlap duration.
Book → look
Invented worked example applying the article’s debugging order. Utterances and timestamps are illustrative; spacing is schematic. No recorded audio is shown.
- Establish intent. At 00:12.180 in this invented trace, the caller says “book the 10 a.m. slot.” Treat this as the example’s reference intent, not as a quote from a recorded call.
- First divergence. At 00:12.910, “book” becomes “look,” with low confidence on the first token. This is the first evidence of intent changing between layers. Start the investigation at hearing.
- Read downstream fairly. At 00:13.240, the model asks for clarification rather than calling the booking tool. Assess that move against what it received: the already changed transcript. No booking tool execution is present in this example.
- Then inspect timing. At 00:14.800, this example assumes TTS starts while the caller is correcting the reply. Delivery adds a second failure. The correction’s start time and playback duration are omitted, so these events cannot establish an overlap duration.
Repair the first divergence, then read again
Choose a synthetic failure, compare each layer with the evidence it received, and apply a local repair. The two-fault case has a second problem waiting downstream.
Invented evidence, ordered by causality rather than measured timestamps. A downstream action can be locally reasonable and still carry an upstream mistake.
01 · Hearing
- Received
- Reference audio intent: book the Tuesday slot.
- Expected at this layer
- Book the Tuesday slot.
- Observed
- Look at the Tuesday slot.
Check the original audio, entity vocabulary, and STT configuration; preserve this utterance as a regression.
- OpenAI Realtime conversations guideDocuments the event flow connecting audio, text, function calls, and session state in a realtime conversation.
- LiveKit turn detection documentationDocuments VAD, endpointing, turn completion, interruption handling, and speech scheduling as separate controls.
- Voice agent interruption handling runbookProvides a logging checklist for user speech state, agent speech state, interruption decisions, playback actions, and recovery.
- Pipecat response_cancel_not_active issueA small but real example of interruption/cancel semantics turning into a fatal voice-session bug.
- Realtime API bot hangs after function callsShows why transcript, tool-call logs, event-loop behavior, and turn state all have to be read together.
Where the fix belongs
Each layer requires a different kind of fix:
- Hearing failures want config and vocabulary work (STT hints, entity lists), not prompts.
- Decision failures want prompt or policy changes, and an eval case proving the change.
- Tool failures want code: timeouts, retries, validation on what comes back.
- Delivery failures want orchestration tuning: interruption thresholds, end-call conditions.
The prompt is often the easiest component to edit, so teams use it to compensate for failures in speech recognition, APIs, or turn timing. Those instructions accumulate until a later prompt change affects behavior that should have belonged to code or configuration.
Preserve the failing call
The investigation ends by capturing the input phrasing, expected tool call or behavior, and an assertion for the suite. A caller's unusual phrasing can recur with another caller, and the transcript preserves the evidence needed to test it again.
The work resembles reading a flight recorder: align the transcript, tool events, and timestamps until the sequence makes sense. Voice failures include mishearing, wrong decisions, slow answers, and interruptions, so the transcript needs the surrounding event trace before it can explain the call.