What Actually Happens When You Call an LLM API

I started sketching this for a client after the third time someone asked me why "the model" got slower at six in the evening. The mental model most teams carry is:

prompt -> model -> answer

That model is fine right up until production, where it explains none of the things that actually happen to you: the 429s, the latency that varies for identical prompts, the bill that doesn't match intuition. A more useful operational model looks like:

prompt
-> gateway (auth, quotas, rate limits)
-> tokenizer
-> router
-> scheduler / queue
-> prefill
-> decode, one token at a time
-> safety checks
-> complete response or streamed response events
-> billing and logs

This is an operational model rather than a description of any provider's private infrastructure. The stages differ by provider, and some boxes describe observable API behavior while others describe conceptual inference work. It is still a better debugging model than treating the endpoint as a single model process because production symptoms can originate in the gateway, scheduler, network, or application loop.

The request path: what crosses the API boundary

What crosses the boundary between your application and the provider, and what evidence comes back?

TakeawayA cache hit reuses eligible prefix work before the request rejoins inference. Complete responses return once; streams return ordered events while generation continues.

Debug from the evidence that crosses that API boundary rather than inferring private provider behavior from a single trace.

The gateway: where 429 shows up

A 429 feels like "the model is overloaded," but the status alone does not establish the cause. A 429 can mean that a documented request, token, or concurrency limit was reached; some providers also use it for temporary service-side throttling or capacity pressure. Read the error details and rate-limit headers before choosing the fix: request-rate limits may want batching or spacing, token-rate limits may want shorter prompts or a different tier, and concurrency limits may want a queue on your side.

The provider request ID matters here. Save it with your own trace ID. If you only log "OpenAI 429" or "Vertex timeout," you have thrown away the one handle support, billing, and incident review can all use later.

The tokenizer: where the bill is decided

Most text-generation APIs turn your text into token IDs before inference, and usage is usually metered in tokens rather than characters. Compression varies: prose, code, and JSON can produce different counts, but which representation costs more depends on the tokenizer and the exact text. Verbose keys, long schemas, and repeated context can make a structured representation costlier than a compact one. When a bill surprises you, inspect the token counts before blaming the nominal prompt length.

The router: why "the model" isn't one thing

The endpoint name you call is a label, not proof that every request reaches one fixed machine. A provider may route work across serving pools, regions, hardware, or model variants, but those decisions are provider-specific and often invisible to callers. Routing is therefore a possible explanation for changed latency or behavior, not a hidden fact you can infer from one trace. If your system silently depends on behavioral quirks, evals will reveal changes. Guesses about private routing will not.

The queue: one reason latency can double

GPU serving can be queue-sensitive in the way all heavily utilized systems are: near saturation, small increases in load can produce large increases in waiting. That's queueing theory, not an AI-specific law. Queueing is one possible cause of a six-o'clock slowdown, but an API trace alone does not prove the provider was at peak load or reveal which internal queue was involved. Treat "the model got slower" as a symptom, then use the timing and provider evidence you actually have.

The practical consequences for anyone integrating:

Log latency in segments, not as one number. Time to first token and time between tokens have different causes and different fixes.
Expect variance, and design timeouts around the tail you observe, not the average you were promised.
Same prompt, different latency is normal, not a bug to file.

Prefill and decode: the two-act structure

Prefill processes the prompt with substantial parallelism and prepares state used during generation, although the exact implementation varies. Autoregressive decode then produces token positions in sequence because each new token depends on earlier output. That sequential dependency limits parallelism across output positions, even though serving systems can still parallelize model computation, batch requests, or use techniques such as speculative decoding. For long responses, decode can dominate end-to-end latency, which makes output length a practical lever.

This split also explains streaming. It is not just a UX flourish: delivering response events as output becomes available can reduce perceived waiting. Streaming does not remove the decode work, token usage, or protocol overhead; it changes when the user can begin reading, not how much generation happened.

Where did the elapsed time go after this illustrative request entered the service?

TakeawayFirst-token generation is only part of the trace. Longer output adds generation work; streaming lets the application receive text while that work continues.

A long prompt and a long output leave different trace shapes, so time to first token and decode time need separate measurements.

Illustrative serial timing model with constant token pacing and immediate delivery. Prefill includes producing the first token; decode supplies the remaining 79. You supply every duration. Real API traces usually cannot isolate queueing from prefill, and network delay, buffering, or hidden reasoning can delay visible text.

Assumed queue wait300 ms

Prefill through first token500 ms

Output length80 tokens

Assumed time between tokens20 ms

Application timeout2,000 ms

Delivery mode

Shared time axis · terracotta rule = application timeout

01 · Queue300 ms

02 · Prefill + first token500 ms

03 · Remaining decode1580 ms

0 ms1238 ms2475 ms

First token generated800 msQueue + prefill, including the first token.

First content would appear800 msStreaming exposes the first generated position.

Modeled completion2380 msStreaming leaves total generation work unchanged.

Partial stream

61 / 80 positions delivered by the earlier of completion or timeout. Filled marks are visible; empty marks are not delivered.

The reader sees a partial stream before the timeout. The application still needs an explicit incomplete-response state and cancellation policy.

Inspect this illustrative trace receipt

{
  "scenario": "illustrative serial inference model",
  "streaming": true,
  "assumedQueueMs": 300,
  "assumedPrefillMs": 500,
  "modeledFirstTokenMs": 800,
  "modeledCompletionMs": 2380,
  "timeoutMs": 2000,
  "visibleTokensAtEnd": 61,
  "outcome": "partial"
}

Actual traces should also preserve request IDs, usage, finish reason, and provider-returned errors. This toy model assumes no retries or tool turns; a token arriving exactly at the deadline is counted as delivered. A client timeout does not prove that provider work or billing stopped.

Prefill and decode: NVIDIA’s inference explanation

Trace rowThe fields that make an LLM request debuggable later

Identity: request ID, tenant, route, provider, model, model version if exposed, prompt version, and tool schema version.

Shape: input tokens, cached input tokens, output tokens, max output requested, streaming on/off, retry count, and finish reason.

Timing: queue wait if known, time to first token, median inter-token latency, total duration, and timeout budget.

Outcome: final status, provider request ID, rate-limit ledger hit if available, safety/refusal signal, and the trace pointer you will need during the incident review.

The exact names do not matter. Keeping the segments separate does.

Tool calls: the application owns the loop

A tool definition gives the model a vocabulary for requesting work; it does not move your custom function into the provider's infrastructure. Who executes the requested tool, and why does the model receive another turn afterward?

TakeawayThe model requests a call; the application validates and executes it. Failure handling is application policy, not automatic recovery.

The application owns the loop: the model proposes a call, the application executes it only after validation, matching output is appended for the next model turn, and application policy decides what happens when execution fails.

Source trailWhat this post is in conversation with

Life of an inference request in vLLMDiscussion of the vLLM request lifecycle, including prefill, decode, KV cache, continuous batching, and cache-aware routing.
How Modern LLM Inference WorksDetailed explanation of prefill/decode mismatch, KV-cache transfer, prefix caching, and goodput under SLA.
Inference Latency: Prefill, Decode, and BatchingClear vocabulary for queueing, TTFT, TPOT/ITL, and end-to-end latency.
OpenAI text generationOfficial reference for Responses requests, instructions, output_text, and why the output array can contain more than message text.
OpenAI rate limitsOfficial reference for interpreting rate-limit errors and the limit information exposed to callers.
Prompt cachingDocuments cached-token accounting and the stable-prefix requirement, making cache behavior observable in an application trace.
OpenAI streaming API responsesOfficial reference for stream=true and server-sent events in the Responses API.
OpenAI Structured OutputsOfficial reference for schema-constrained output and streaming structured data safely.
OpenAI function callingOfficial reference for tool calls, tool outputs, and why tool orchestration is an application loop.
Mastering LLM Techniques: Inference OptimizationExplains KV-cache memory, batching, and decode constraints from the serving side of the system.

What I actually changed after internalizing this

We cap output lengths where a long answer is not the product. We log provider, model version, and segmented latency on every request. We classify 429s from the returned evidence instead of assuming one cause, and we test queueing as one hypothesis when latency drifts.

The diagnostic order matters. First ask which limit or transient condition the response identifies. Then ask which segment got slower. Then ask whether input or output shape changed. Only after those questions should "the model got worse" become the leading theory.

These practices come from treating the API as a distributed service with observable boundaries, variable load, and failure modes outside the model weights.

What Actually Happens When You Call an LLM API

The request path: what crosses the API boundary

Which boundary does the request cross, and what returns?

The gateway: where 429 shows up

The tokenizer: where the bill is decided

The router: why "the model" isn't one thing

The queue: one reason latency can double

Prefill and decode: the two-act structure

Where did the elapsed time go?

Move the delay; watch the trace change

Partial stream

Tool calls: the application owns the loop

Who executes the tool, and why is there another model request?

What I actually changed after internalizing this

Share This Entry

Related Topics

Discussion