LLM providers commonly charge more for output tokens than input tokens. The difference follows, in part, from how serving systems process a prompt and generate a response. Understanding those two phases makes output limits, prompt caching, and streaming easier to reason about.
One pass in, one token at a time out
When your request arrives, the prompt is processed in a single parallel pass. Every input token flows through the model together, the GPU does what GPUs love (large batched matrix work), and the per-token cost is comparatively small. This phase is called prefill.
During generation, each new token depends on the previous output tokens. Serving systems still batch requests and parallelize model computation, but they cannot generate all positions in a response in one pass. Longer outputs therefore consume repeated decode steps and hold serving resources for longer.
The model also caches attention keys and values for the tokens processed so far, and that state occupies GPU memory for the duration of a request. Long contexts affect memory occupancy as well as compute, which can reduce the number of requests a serving system handles concurrently.
Inference workbench
A crowd in; a dependency chain out
Choose a short or longer output and follow the token work. The lower lane remains present while generation progresses.
Scenario
Illustrative token counts. Diagram distance and playback speed are not measured time or a price estimate.
Sequence
- Parallel inputCurrent
- Output 1Upcoming
- Output 2Upcoming
- Output 3Upcoming
- Output 4Upcoming
Parallel input
Prompt positions can be processed together during prefill. The final prompt position supplies the logits for output token 1, revealed in the next step. The marks represent tokens, not words; no tokenizer or model is being run here.
Read the full explanation
4 output tokens
Illustrative token counts. Diagram distance and playback speed are not measured time or a price estimate.
- Parallel input. Prompt positions can be processed together during prefill. The final prompt position supplies the logits for output token 1, revealed in the next step. The marks represent tokens, not words; no tokenizer or model is being run here.
- Output 1. Reveal the first token sampled from the prefill output. This reveal is not an extra decode forward pass. Later tokens require processing the preceding generated token.
- Output 2. Generate output token 2 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 3. Generate output token 3 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 4. Generate output token 4 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
8 output tokens
Illustrative token counts. Diagram distance and playback speed are not measured time or a price estimate.
- Parallel input. Prompt positions can be processed together during prefill. The final prompt position supplies the logits for output token 1, revealed in the next step. The marks represent tokens, not words; no tokenizer or model is being run here.
- Output 1. Reveal the first token sampled from the prefill output. This reveal is not an extra decode forward pass. Later tokens require processing the preceding generated token.
- Output 2. Generate output token 2 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 3. Generate output token 3 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 4. Generate output token 4 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 5. Generate output token 5 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 6. Generate output token 6 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 7. Generate output token 7 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 8. Generate output token 8 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
4 output tokens
Illustrative token counts. Diagram distance and playback speed are not measured time or a price estimate.
- Parallel input. Prompt positions can be processed together during prefill. The final prompt position supplies the logits for output token 1, revealed in the next step. The marks represent tokens, not words; no tokenizer or model is being run here.
- Output 1. Reveal the first token sampled from the prefill output. This reveal is not an extra decode forward pass. Later tokens require processing the preceding generated token.
- Output 2. Generate output token 2 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 3. Generate output token 3 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 4. Generate output token 4 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
8 output tokens
Illustrative token counts. Diagram distance and playback speed are not measured time or a price estimate.
- Parallel input. Prompt positions can be processed together during prefill. The final prompt position supplies the logits for output token 1, revealed in the next step. The marks represent tokens, not words; no tokenizer or model is being run here.
- Output 1. Reveal the first token sampled from the prefill output. This reveal is not an extra decode forward pass. Later tokens require processing the preceding generated token.
- Output 2. Generate output token 2 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 3. Generate output token 3 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 4. Generate output token 4 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 5. Generate output token 5 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 6. Generate output token 6 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 7. Generate output token 7 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
- Output 8. Generate output token 8 using the preceding context. Cached attention state remains available alongside decode; it is not a separate waiting phase. This is a simplified autoregressive path, without speculative decoding.
Change the shape of the work
Vary the prompt, shared prefix, and generated length. Watch what disappears from prefill and what stays in the resident context.
Illustrative accounting: one uncached input position costs one work unit. You set the output weight. These are assumptions, not prices, GPU measurements, or a serving-time forecast. Cache reuse is assumed available.
Output dominates under your selected weight. Try halving the generated length. Streaming changes when a reader sees output; it would leave these totals unchanged.
What providers do about it, and why you see the side effects
Providers use techniques such as continuous batching to interleave decode steps from many requests and keep GPUs occupied. The resulting latency still varies with queue depth, request shape, and the other work being served at the same time.
Prompt caching provides another lever. When requests share a long prefix such as a system prompt or tool schema, a provider can reuse computed state instead of repeating the entire prefill. Discounted cached-input rates expose some of that reuse to customers.
The operational catch is that caching is literal. A timestamp, request ID, randomized example order, or user-specific paragraph placed near the top can turn a reusable prefix into a miss. The system may be logically equivalent and economically different. Prompt organization is therefore not only readability. It is cache-key design.
- 01Stable prefix firstSystem prompt, tool schemas, output contract, and examples stay byte-for-byte stable at the top.Cacheable.
- 02Volatile context laterRetrieved chunks, user-specific facts, timestamps, and conversation tail come after the shared prefix.Pay full price only for what changes.
- 03Deterministic serializationKeep object key order and schema wording stable so logically identical prompts are also textually identical.A reordered JSON object is a cache miss.
What this should change in how you build
Practical consequences, in the order they've saved me money:
- Cap output length anywhere prose isn't the product. Output is the expensive direction and the slow one. An extraction task that returns a paragraph of polite framing around the JSON is paying premium rates for filler. Asking for terseness is a cost optimization, not a style preference.
- Structure long prompts for cache hits. Stable prefix first (system prompt, schemas, examples), volatile content last. A reordered prompt that breaks prefix-sharing silently turns cached rates back into full rates.
- Don't pad max-token limits "to be safe." Generous ceilings invite generous answers, and some serving stacks make scheduling decisions off your declared maximum. Set limits near what correct output actually needs.
- Watch the verbosity of structured output. Schema design is cost design: key names are tokens, nesting is tokens, and a chatty schema multiplies across every request you'll ever make.
- Stream by default for anything user-facing. It doesn't make generation faster, but it makes the sequential wait honest, and perceived latency is the latency users act on.
- Transformers cachingThe workbench simplifies autoregressive generation: prior attention keys and values are reused as the output grows. It is not a measured cost or latency model.
- Prefix cachingPractical explanation of KV reuse across shared prompt prefixes and where cache hit rates come from.
- How prompt caching worksReadable walkthrough of paged attention, block hashing, and why stable prefixes matter.
- Disaggregated prefill and decode on SageMaker HyperPodProvider-side view of the same prefill/decode split and why long prompts disturb decode latency.
- OpenAI prompt cachingOfficial reference for cached-token accounting and why stable prompt prefixes become measurable cost signals.
- Efficient Memory Management for Large Language Model Serving with PagedAttentionThe vLLM paper explains why KV-cache memory management is central to serving throughput, not an implementation footnote.
I spend output tokens where the generated text is itself the product. Extraction and classification paths use tighter ceilings, stable prompt prefixes, and compact schemas because verbosity there adds cost without helping the user.
Two consequences of the mechanism
A large context window can become a memory constraint because the serving system keeps cached state for context tokens while generating the response. That state competes with other requests for GPU memory.
The declared maximum output can also affect scheduling or memory planning in some serving stacks even when a response stops early. For that reason, I set the ceiling near the longest correct answer instead of treating it as a harmless safety margin.