Why Is the First Token of Large Models Always the Slowest?

Next time you send a long document to a large language model (LLM), pay attention to an experience that isn’t rare but is rarely explained: it takes several seconds for the first character to "appear," but once it does, the subsequent characters stream out faster and faster. This isn’t due to network jitter; it’s caused by the two-stage division of labor within the model itself: prefill and decode.

Two Stages: Process the Prompt First, Then Generate Token by Token

When the model receives your input, it first enters the **prefill** stage. All tokens in the prompt are processed in parallel through the network, computing attention layer by layer, to produce the "state required to generate the first token" in one go. This stage is about "doing many things simultaneously." While the total computational load is high, the wall-clock time is relatively short.

Afterward, it enters the **decode** stage. Generating the second token depends on the result of the first, the third depends on the second, and so on, forcing a sequential, token-by-token progression. It’s like writing a mathematical proof: each step must wait for the previous conclusion to be established. Adding more machines won’t help parallelize this process.

The bottlenecks in these two stages differ. Prefill is bottlenecked by **compute power**; the GPU is busy performing matrix multiplications under heavy load. Decode is bottlenecked by **memory bandwidth**. Although only one new token is computed per step (a small amount of computation), each step requires fetching the model weights and the KV cache of all previous tokens from VRAM. With a 32K context window, the cost per step in decode is an order of magnitude higher than the average cost per token in prefill.

KV Cache: Avoid Recalculation, Rely on Caching

Without caching, generating each new token would require recalculating its attention with all preceding tokens. Generating $n$ tokens would require approximately $n \times (n-1) / 2$ pairwise calculations, leading to a direct quadratic explosion. Therefore, every Transformer layer includes a cache called the **KV cache**: the Key and Value vectors computed for each token at each layer are stored. When generating the next token, only the K/V for the new token is calculated, while the old ones are retrieved directly.

The trade-off is that VRAM usage expands linearly with the context length. Let’s calculate this for a hypothetical 8B parameter model: 32 layers, 8 KV heads per layer, 128 dimensions per head, stored in fp16. For a single token, each layer stores $2 (K+V) \times 8 \times 128 \times 2 \text{ bytes} = 4 \text{ KB}$. Across 32 layers, this amounts to 128 KB per token. For a 32K context window, the KV cache occupies approximately 4 GB. This is the direct source of the saying "long contexts eat up VRAM," and it’s why techniques like GQA (Grouped Query Attention) and MQA (Multi-Query Attention) target this area—by reducing the number of KV heads from 32 to 8, cache usage is cut by three-quarters.

Three Actionable Takeaways

1. **TTFT and TPOT are two distinct metrics.** Time to First Token (TTFT) reflects the speed of prefill, while Time Per Output Token (TPOT) reflects the speed of decode. When users complain that "responses are slow," distinguish which metric is at fault: scenarios with long prompts are usually bottlenecked by TTFT, while scenarios with long outputs are usually bottlenecked by TPOT. Lumping them together as "high latency" leads to misguided optimization efforts.

2. **Place stable content at the beginning.** For prefixes shared across multiple requests, placing them earlier increases the probability of hitting and reusing the KV cache directly. Therefore, putting system prompts, tool descriptions, and few-shot examples at the start of the prompt, while placing the user’s variable input at the end, is an engineering decision with practical significance, not just a formatting habit.

3. **Don’t fill up the context window unnecessarily.** It’s not that the model can’t remember; it’s that lookup is expensive. For every token generated, the model must perform a lookup against the KV cache of all preceding tokens. Stuffing 100,000 words into the context when 90,000 are irrelevant to the question means every token’s latency includes the "transport cost" of moving that massive amount of data. The correct approach is **retrieve-then-feed**: use a vector database to retrieve candidate segments, use a reranker to keep the top-k, and then feed only those relevant segments to the model, spending tokens only where necessary.

The next time you start a long-document Q&A session in the evening, watching that solitary first character slowly emerge, remember that you aren’t just waiting for it to "think" of that one word. You are waiting for the GPU to parallel-process your entire prompt, write several gigabytes of cache into VRAM, and only then begin to write.