AI Inference Optimization Deep Dive: Quantization, KV Cache, and Speculative Decoding — Why Speed Beats Accuracy

Last week I was chatting with a friend who does ML, and he said something that stuck with me: "Everyone is racing to scale up model parameters, but what actually creates a gap in production environments is inference speed."

He's right. When GPT-4o launched, people were amazed not by how much smarter it was than GPT-4, but by how much faster and cheaper it was. Inference efficiency has quietly become the main battlefield of AI competition.

Why Does "Fast" Matter More Than "Accurate"?

This sounds counterintuitive at first. But think about it — a model that's 98% accurate but takes 30 seconds to respond, versus one that's 95% accurate but gives you an answer in 500 milliseconds. Which one would you use in actual work?

In most scenarios, you'd choose the fast one.

This isn't saying accuracy doesn't matter. It's saying: once accuracy clears a "good enough" threshold, speed and cost become the more critical variables. This rule has been repeatedly validated in industry.

So over the past two years, inference optimization has suddenly become a hot direction. Not because people don't want smarter models, but because "how to make existing models run faster and cost less" has a higher return on investment.

Quantization: The Art of Compressing Models

The most common inference optimization technique is quantization.

Simply put: neural network weights are typically stored as 32-bit floating point numbers (FP32). Quantization compresses these numbers into smaller formats — 16-bit (FP16), 8-bit (INT8), or even 4-bit (INT4).

What's the cost? Slight precision loss. But in practical testing, compressing from FP32 to INT8 is barely perceptible in model performance, while inference speed can improve 2-4x and memory usage cuts in half.

At SFD Lab, when we run local Ollama inference, the most tangible feeling is the difference quantization makes. The same Qwen2.5-32B model: the Q4_K_M quantized version runs smoothly on a 96GB Mac Studio, but try running the FP16 version and memory immediately runs out. Quantization isn't a "make-do" solution — in many cases it's the only viable option.

KV Cache: Stop Making the Model Redo Work

There's another major component of inference: KV Cache (Key-Value Cache).

When a model is generating long text, for every token it generates, it needs to "look back" at all previous content. If this is recalculated every time, the cost is O(n²) — computation explodes exponentially as text length grows.

KV Cache's logic: cache the previously computed attention key-value pairs, and read directly next time instead of recomputing. This optimization has enormous effect in long-text generation scenarios, improving response speed by several to over ten times.

The problem is that KV Cache itself is memory-intensive. When serving many user requests simultaneously, each request must maintain its own Cache, and memory pressure quickly builds. This is why there's now a lot of research on "KV Cache compression" and "cross-request KV Cache sharing."

Speculative Decoding: Parallel Betting

There's also a relatively novel direction called speculative decoding.

The idea is interesting: use a small model to quickly "guess" the next several tokens, then use the large model to "verify" whether these guesses are correct. If the guesses are right, accept them directly, skipping the large model's token-by-token generation step. If wrong, restart from where things went wrong.

Because the small model runs quickly, and verifying multiple tokens with the large model costs less than generating them one by one, the overall inference speed can improve 30-50%, while output results are completely identical to using the large model directly.

This approach requires "the small model can guess correctly most of the time" — and in actual language generation scenarios, this hit rate is indeed quite high, because many token continuations are highly predictable ("I had a bowl of" → the next word is almost certainly some kind of food).

Batching and Continuous Batching

In inference services serving multiple users, batching is another key factor.

Early inference frameworks used static batching: wait to accumulate a batch of requests before computing together. The problem: some requests finish quickly, some slowly, and fast ones wait for slow ones, wasting GPU resources.

Continuous batching solves this: whoever finishes generating exits, and newly arrived requests are directly inserted into the batch to continue. vLLM is one of the frameworks that pushes this to the extreme — in high-concurrency scenarios, throughput improvement can reach 10x or more.

MoE: Activate On-Demand, Avoid Unnecessary Cost

Finally, an architecture-level optimization: Mixture of Experts (MoE).

Traditional Dense models run all parameters for every inference. MoE's idea: split the model into many "expert" sub-networks, activating only a few (typically 2-8) for each inference, with a router deciding which experts to use for the current input.

Mixtral 8x7B is a classic MoE architecture: total parameters look like 46B, but only 12-13B are actually running each time, with inference cost similar to a 13B Dense model but performance close to a genuine 70B model.

This "on-demand activation" design lets models maintain large capacity while significantly improving inference efficiency.

Where Technology Races, Products Find Opportunity

When writing this, I was thinking about a question: inference optimization is so important, so why is user awareness so low?

Because most optimization happens "behind the scenes" — users just feel "seems faster now," without knowing what happened behind the curtain.

But for those truly building products with AI, understanding these mechanisms matters. It determines which inference framework you choose, what quantization precision you use, how you design request batching logic. These decisions directly affect your costs and user experience.

AI competition has partially shifted from "who has smarter models" to "who has more efficient systems." The latter demands engineering capability no less than the former.