The Economics of LLM Inference: What a Token Really Costs

A single API call to a large language model looks like it costs a fixed amount: some number of cents per million tokens, multiplied by however many tokens went in and came out. That number on the pricing page is real, but it hides a much messier truth underneath. The cost of producing that response was assembled from at least half a dozen different resource costs — some paid once per request, some paid once per token, some paid continuously whether or not anyone is using the GPU at that moment — and the ratio between them shifts depending on prompt length, output length, batch size, and how the serving software is built.

Understanding that breakdown matters if you're building a product on top of an LLM API, running your own inference stack, or just trying to figure out why a "cheap" model can end up costing more than an expensive one for your specific workload. This piece walks through what actually happens between a request landing on a GPU and a response streaming back, and why the industry has spent the last two years re-architecting serving infrastructure around that breakdown.

Inference is two different jobs wearing one name

"Running inference" sounds like a single operation, but every LLM request is really two distinct computational phases stitched together, and they have almost opposite performance characteristics.

Prefill is the phase where the model processes your entire input prompt at once — the system instructions, the conversation history, the document you pasted in, the tool definitions. The model runs this whole sequence through its layers in a single parallel pass to build up an internal representation of everything it has been given. This is compute-bound: it uses a large fraction of the GPU's raw arithmetic throughput, and it takes roughly the same amount of wall-clock time whether the GPU is otherwise idle or fully loaded, because the work is one big parallelizable matrix operation.

Decode is the phase where the model generates output, one token at a time. To produce token N+1, the model needs to attend back over every token that came before it — the entire prompt plus everything it has generated so far. Because each new token depends on the one just produced, this phase is inherently sequential: you cannot generate token 50 before token 49 exists. Each individual decode step touches comparatively little new computation, but it has to repeatedly read a large amount of state from memory. That makes decode memory-bandwidth-bound rather than compute-bound — the GPU's math units sit mostly idle, waiting on data to arrive from memory.

This split is the single most important fact in LLM inference economics, because it means the two phases compete for different resources. A serving system optimized for one is often wasting the other.

The KV cache: the memory bill nobody put on the pricing page

The mechanism that makes decode possible without redoing prefill work at every step is the KV cache (key-value cache). Once the model has computed the attention keys and values for every token in the prompt and every token it has generated so far, it stores them in GPU memory rather than recomputing them on each new step. Without this cache, generating a 1,000-token response would mean re-processing the entire growing context 1,000 times over; with it, each new token only needs to compute its own keys and values and attend over what's already cached.

That cache is not free. Its size scales with the number of layers in the model, the number of attention heads, the hidden dimension, and — critically — the total sequence length (prompt plus generated tokens so far). For a long conversation, a large system prompt, or a document-heavy RAG pipeline, the KV cache for a single request can occupy gigabytes of GPU memory. Multiply that by dozens or hundreds of concurrent users, and KV cache memory — not model weights — often becomes the binding constraint on how many requests a GPU can serve simultaneously.

This is why "how much memory does the model need" and "how much memory does serving the model need" are different questions. Model weights are fixed and shared across all requests. KV cache is per-request, grows for the duration of the conversation, and directly limits concurrency: a GPU that could technically serve 200 short requests at once might only manage 20 requests with long contexts before it runs out of memory for their caches.

Why this matters right now

For most of the last few years, production LLM serving ran prefill and decode on the same GPU, back to back, for every request. That's simple to build, but it's wasteful: a GPU busy running the compute-heavy prefill for one user sits underutilized on memory bandwidth, while a GPU stuck decoding for a dozen users has spare compute capacity going unused. Batching requests together helps, but a long prefill for a new request arriving mid-batch can stall the decode steps for everyone else already generating — a phenomenon serving engineers call head-of-line blocking.

The current shift is to stop treating prefill and decode as one job. Serving stacks are increasingly splitting them onto separate GPU pools — a set of GPUs dedicated to prefill, tuned for maximum compute throughput on incoming prompts, and a separate set dedicated to decode, tuned for memory bandwidth and high concurrency. When a request arrives, its prompt is processed on a prefill node, the resulting KV cache is handed off to a decode node, and generation proceeds there independently. This disaggregation lets each pool be sized, and even use different hardware, for the job it's actually doing, instead of forcing one GPU generation to be simultaneously good at two opposing workloads.

Alongside that split, serving stacks are also offloading KV caches to flash storage rather than keeping every cache resident in GPU memory. GPU high-bandwidth memory is fast but scarce and expensive; flash storage is far cheaper per gigabyte, if slower to access. Because a conversation's KV cache doesn't need to sit in GPU memory during the gaps between a user's messages, systems can evict it to flash when idle and reload it when the conversation resumes — trading a small latency hit on reactivation for a large increase in how many concurrent conversations a fixed GPU fleet can support. Combined, prefill/decode disaggregation and flash-backed KV cache offload are the two biggest architectural levers serving teams are pulling to bring the effective cost per token down without waiting for new chips.

What actually drives the per-token price you pay

The published price per million tokens on any provider's pricing page is the output of all of the above, compressed into a single number and averaged across the provider's own traffic mix. A few structural factors explain why that number looks the way it does, and why it differs so much between models and providers.

1. Model size and architecture. A larger model needs more FLOPs per token in prefill and more memory bandwidth per token in decode. Mixture-of-experts architectures change this calculus by activating only a subset of total parameters per token, which is part of why some very large models are priced closer to mid-sized dense models.
2. Batching efficiency. Serving many requests together lets the compute-heavy parts of a GPU's work amortize across users, which is straightforward for prefill and harder for decode because requests finish at different times. Continuous batching techniques, where new requests join and finished requests leave a running batch without waiting for a fixed-size group to fill or empty, materially improve GPU utilization and therefore lower cost per token.
3. Context length. Both prefill cost and KV cache memory scale with total context. A request with a 100,000-token prompt is fundamentally more expensive to serve than one with a 1,000-token prompt, even producing the same length of output — which is why providers increasingly price long-context usage differently, and why prompt caching (reusing a previously computed prefill for a repeated prefix) has become a standard cost lever.
4. Output length. Because decode is sequential, output tokens are more time-expensive than input tokens per token generated, which is reflected in the output-token price on most providers' pricing pages running several times higher than the input-token price.
5. Hardware generation and utilization. Newer accelerators offer more memory bandwidth and compute per dollar, but only if serving software can keep them busy. A well-optimized serving stack on older hardware can beat a naive deployment on newer hardware.
6. Idle capacity and reserved buffer. Providers must keep spare GPU capacity available to absorb demand spikes and avoid queuing. That reserved, sometimes-idle capacity is a real cost that gets folded into the average price, not a cost that only appears when the GPU is actively computing.

Practical implications for teams building on LLMs

None of this is purely academic if you're shipping a product that calls an LLM API. The economics above translate directly into decisions you can make.

• Prompt caching is the highest-leverage lever available to API consumers. If your application repeatedly sends the same system prompt, tool definitions, or long reference document as a prefix, structuring requests so that prefix is cached avoids paying full prefill cost on every call. This is frequently a 5-10x cost reduction on the cached portion.
• Output length costs disproportionately. Because decode is the sequential, memory-bound phase, trimming unnecessary verbosity in generated output (via prompting, format constraints, or lower reasoning-effort settings) saves more than trimming an equivalent amount from the input side.
• Context management is a cost control, not just a quality control. Aggressively pruning irrelevant history, summarizing old turns, or retrieving only the relevant passage instead of pasting a whole document reduces both the KV cache footprint and the prefill bill on every subsequent turn of a conversation.
• Batch-friendly workloads should be batched. Asynchronous or non-latency-sensitive workloads (bulk classification, offline summarization, evaluation runs) are usually eligible for discounted batch processing, because the provider can schedule them into whatever slack capacity exists rather than reserving latency-sensitive headroom for them.
• Model choice should be matched to phase-sensitivity. A workload dominated by long documents and short answers is prefill-heavy; a workload that generates long structured output from short prompts is decode-heavy. The "best" model or provider for one shape of workload is not automatically the best for the other, even at the same headline price per token.
• Latency and cost are coupled, not independent. Requesting faster response times (through premium latency tiers, dedicated capacity, or higher-priority queuing) generally means paying for reserved GPU headroom that would otherwise be shared, so the fastest option and the cheapest option are rarely the same choice.

Limitations and open questions

The economics described here are directionally solid but genuinely hard to observe from outside a serving provider. A few caveats are worth holding onto.

Published per-token prices are averages, not marginal costs. A provider's price reflects a blend of traffic across many customers, prompt shapes, and load conditions; your specific workload's actual cost to serve could be meaningfully above or below that average, and you generally have no visibility into which. This makes it difficult to reason precisely about whether a given optimization (say, restructuring prompts for better caching) will move your bill by 5% or 40% — the honest answer is usually "test it and measure," not "calculate it from first principles."

There is also no standardized way to compare serving efficiency across providers. Two providers running the same open-weight model at the same published price may have very different actual margins, GPU utilization, and latency guarantees, because the serving stack — batching strategy, disaggregation, cache offload, hardware generation — is invisible from the API surface. Pricing pages tell you what you pay; they don't tell you what it cost to serve you, which is exactly the gap that makes this a genuinely evolving field rather than settled engineering.

Finally, energy and hardware supply constraints sit underneath all of this and are largely exogenous to any individual serving optimization. GPU availability, power delivery to data centers, and high-bandwidth memory supply all shape the floor on inference costs in ways that clever serving software can't fully offset — architectural efficiency gains buy time and headroom, but they don't eliminate the underlying capital and energy intensity of running frontier-scale models at volume.

What to watch next

A few trends are likely to keep reshaping the cost curve over the next couple of years:

• Wider adoption of prefill/decode disaggregation as a default serving pattern rather than a specialized optimization only the largest providers use, as the open-source serving frameworks that implement it mature.
• Cheaper, faster tiers of memory between GPU HBM and traditional flash storage, purpose-built for KV cache offload rather than general storage, narrowing the latency penalty of eviction.
• More granular, usage-shaped pricing — separate rates for cached versus uncached tokens, batch versus real-time, and long-context versus short-context requests are already appearing and are likely to become the norm rather than the exception.
• Specialized inference hardware built around the memory-bandwidth-bound nature of decode specifically, rather than general-purpose accelerators originally designed for training workloads.
• Smaller, more efficient models absorbing more production traffic as routing systems get better at sending easy queries to cheap models and only escalating genuinely hard queries to frontier-scale models, changing the effective blended cost of running an AI product even where per-model pricing stays flat.

FAQ

Why does output cost more per token than input on most LLM APIs?

Generating output happens in the decode phase, which is sequential and memory-bandwidth-bound — each token requires a full pass over the growing context before the next one can be produced. Processing input happens in the parallelizable prefill phase, which uses GPU compute far more efficiently. That structural difference in GPU utilization is why output tokens are typically priced several times higher than input tokens.

What is the KV cache and why does it matter for cost?

The KV cache stores the attention keys and values the model has already computed for a conversation, so it doesn't need to recompute them at every new token. It's the main reason decode is fast, but it consumes GPU memory proportional to context length, and that memory footprint — not model weight size — is often the real limit on how many concurrent users a GPU fleet can serve.

Does a longer conversation cost more even if I only send a short new message?

Yes. Every turn re-processes (or re-uses a cached version of) the full conversation history as context, and the KV cache for that history has to be resident or reloaded for the model to generate a response. Longer running conversations carry rising prefill and memory costs even when each individual new message is short.

What is prefill/decode disaggregation?

It's a serving architecture that runs the compute-heavy prefill phase and the memory-bandwidth-heavy decode phase on separate pools of GPUs, tuned differently for each job, instead of running both phases on the same GPU for every request. It reduces resource contention between the two phases and improves overall throughput per GPU.

Why do some providers charge less for cached tokens?

When a request reuses a prompt prefix that was already processed in a recent request, the serving system can skip re-running prefill for that portion and reuse the previously computed KV cache instead. Since that skips the expensive compute step entirely, providers pass some of that savings back as a lower price for the cached portion of the request.

Is a cheaper model always cheaper to run for my use case?

Not necessarily. A lower per-token price doesn't account for how prefill-heavy or decode-heavy your specific workload is, how well the provider's serving stack batches your traffic pattern, or how much of your context is cacheable. Two models with different sticker prices can end up costing similarly — or invert — depending on your prompt and output shapes.

Will inference costs keep dropping over time?

Historically yes, driven by more efficient model architectures, better serving software (disaggregation, caching, batching), and newer hardware generations. But the rate of decline is not guaranteed to continue at the same pace, since it's bounded by real constraints in energy availability, chip supply, and memory bandwidth that software optimization alone can't remove.

Teams trying to actually engineer their prompts, context strategy, and model routing around these cost dynamics — rather than guessing at what moves the bill — can get hands-on help from Woyce Technologies.