Prefix Caching: Stop Paying Repeatedly for the Same System Prompt

You send thousands of requests to LLM APIs every day, each carrying the same system prompt and the same few-shot examples. The model doesn’t know they’re identical—unless you use prefix caching, every part gets recomputed from scratch. Prefix caching (also called context caching in various products) doesn’t change the model or the answers; it does just one thing: it saves the intermediate K/V results computed for the repeated prefix of a request, so the next request can reuse them.

What Exactly Is Being Cached

It’s not the final answer. It’s the key/value tensors computed for each token of the prompt in every attention layer during autoregressive generation. When generating the first token, the main cost of a long prompt lies here. Prefix caching stores these results as KV blocks. If the next request matches byte-for-byte from the start, it directly references these blocks instead of recomputing.

This also explains its hard constraint: matching starts from the very first byte and proceeds sequentially. Add an extra space or remove a newline in the system prompt, and everything from that point onward fails to hit the cache.

The Most Important Engineering Rule: Fixed First, Variable Last

Cache hit rates are almost entirely determined by the structure of your prompt:

- Place the system prompt at the very beginning and keep it stable over time. You can modify it, but append changes to the end so that all preceding stable content continues to hit the cache.

- Put variable content—such as user input, retrieved documents, or current timestamps—at the end. Place timestamps at the very tail, not at the beginning like “Today is 2026-08-28.”

- In retrieval scenarios, place stable few-shot examples before the retrieved results. This allows the same few-shot block to be reused across multiple rounds.

A common mistake in RAG systems is randomly shuffling “user question + retrieved chunks” before concatenating them into the prompt. This scrambles the prefix starting from the fourth token, rendering the cache useless.

Where the Benefits Show Up: Three Key Metrics

1. **Time to First Token (TTFT)**: The prompt processing phase accounts for the bulk of the initial response latency. For the same 4k-token prefix, hitting the cache can often reduce TTFT to less than 20–30% of a cache miss, depending on the model and concurrency.

2. **Input Costs**: Major cloud providers charge only about 10% for input tokens that hit the cache. With a 2k-token system prompt and tens of thousands of daily requests, the savings add up to real money day by day.

3. **Dispatch Efficiency**: Requests sharing the same prefix keep their KV blocks in the same GPU memory, allowing the engine to schedule them into the same continuous batch, thereby increasing overall throughput.

Let’s crunch some numbers: A customer service bot has a system prompt plus persona settings totaling 1,500 tokens, with 20,000 daily requests. The input price is $3 per million tokens. Without caching, the daily cost for this prompt is 1,500 × 20,000 ÷ 1,000,000 × 3 = $90. With a 90% cache hit rate, only 10% is billed at full price, costing $18 per day. This creates an annual savings potential of roughly $31,000. If your system prompt is under 200 tokens, the primary value of prefix caching isn’t cost reduction but TTFT improvement—a few hundred milliseconds of reduced latency has a far more noticeable impact on chat product experience than the bill does.

Three Common Pitfalls

- **Expiration happens faster than you think**. KV blocks are evicted after being idle for a few minutes. If your traffic pattern is “500 requests in ten minutes followed by a three-hour gap,” you’ll encounter many cache misses—not because the cache is broken, but because it expired.

- **Switching models = full recomputation**. Caches are organized by model weights. When you A/B test or switch models, hit rates drop to zero for a minute, causing a spike in TTFT. This is normal.

- **Don’t just look at the total usage bill**. The `cached_tokens` field in the response reflects the true hit rate. If the weekly average drops to 30–40%, someone likely added content to the beginning of the prompt. Treat this as an alert metric.

Three Steps to Implementation

First, review your existing prompts from front to back and reorder them according to “never changes → occasionally changes → changes every time.” This step costs nothing and usually yields immediate results. Next, ensure you’re using native cache-supported endpoints rather than building your own Redis solution (RAG semantic caching is different—it caches entire answers, so don’t confuse the two). Finally, monitor the `cached_tokens / prompt_tokens` ratio; if it falls below a threshold, it means someone has modified the shared prefix.

Prefix caching doesn’t improve model capabilities. It simply ensures you pay once and wait once for things you’ve said eight hundred times.