Advanced KV Cache: From Static Caching to Dynamic Compression

In the previous article, we discussed the basic principles of KV Cache. However, as context windows expand from 32k to 1M tokens or even more, simple caching strategies can no longer meet the demand. The core conflict now is: the model needs to remember all history, but GPU memory cannot hold it all.

The Necessity of Dynamic Compression

If a conversation contains 100,000 tokens, the KV Cache will occupy tens of gigabytes of VRAM. This means a single GPU can serve only a very small number of users. To address this issue, the industry has begun exploring "selective forgetting" and "dynamic compression."

Core Technical Approaches

1. H2O (Heavy Hitter Oracle):
   Research has found that only a tiny fraction of tokens (Heavy Hitters) in the Attention Map have a decisive impact on the final output. H2O compresses the cache size by 5x–10x without significantly reducing accuracy by tracking the cumulative attention weights of each token in real time and evicting KV pairs with low contribution.

2. StreamingLLM (Attention Sink):
   This is a surprising discovery: when processing long texts, models assign huge attention weights to the first few tokens (even if they are meaningless spaces or punctuation), known as "Attention Sinks." By retaining the initial few tokens plus the most recent tokens in a sliding window, StreamingLLM enables infinite-length streaming inference within fixed memory limits without crashing.

3. Quantized KV Cache (INT8/FP8):
   This approach quantizes the KV Cache from FP16 to INT8 or FP8. It directly halves memory usage. Since KV Cache is relatively insensitive to precision, the performance loss is negligible.

Implications for Architecture Design

For developers building AI applications, this means we can no longer treat LLMs as simple "black-box APIs." When designing long-conversation systems, you should consider:

• Anchoring Key Information: Use prompts to guide the model to place important information at the beginning (leveraging Attention Sinks).
• Segmented Summarization: Do not rely solely on the model's native long-context capability. Instead, implement a loop at the application layer: recursive summarization $\rightarrow$ context update $\rightarrow$ cache clearing.

Summary

The evolution of KV Cache has moved from "full storage" $\rightarrow$ "efficient organization (PagedAttention)" $\rightarrow$ "selective retention." The future trend is to enable models to mimic human memory, featuring a rapidly updated short-term memory and a compressed long-term memory.