Speculative Decoding: A "Fast-Slow" Collaboration Mechanism to Accelerate LLM Inference

In the inference process of Large Language Models (LLMs), the most persistent bottleneck has always been the **Token-by-Token** autoregressive generation mode. No matter how powerful the model is, generating 100 tokens requires 100 complete forward passes through the model. For models with massive parameter counts (such as Llama-3-70B or GPT-4), this results in extremely high latency.

To break this bottleneck, academia and industry have introduced an ingenious solution: **Speculative Decoding**.

Core Logic: Fast Guessing, Slow Verification

The core idea behind speculative decoding is that **not all token predictions are equally difficult.**

In a sentence, what follows "I like to eat" is highly likely to be "apples" or "rice." Such simple predictions do not require a 70-billion-parameter model; they can be handled by much smaller models. It is only when complex logical reasoning or specialized knowledge is involved that the full power of a large model is needed.

Speculative decoding achieves this by introducing collaboration between a **small Draft Model** and a **large Target Model**:

1. **Speculation Phase:** The small model (which is extremely fast) consecutively predicts the next $K$ tokens (e.g., 5 tokens). Since the small model is lightweight, the time required for these $K$ forward passes is far less than the time for a single forward pass of the large model.

2. **Verification Phase:** The large model takes these $K+1$ tokens as input at once and computes their probability distributions in parallel.

3. **Acceptance and Correction:** The large model checks whether the small model's predictions fall within an acceptable range of its own probability distribution.

- If the first 3 are correct but the 4th is wrong $\rightarrow$ Accept the first 3, correct the 4th, and discard all subsequent speculative results.

- If all are correct $\rightarrow$ Output all $K+1$ tokens at once.

Key Trade-offs in Engineering Implementation

When deploying speculative decoding in practice, engineers face three core challenges:

1. Selection of the Draft Model (The Draft Model Gap)

If the small model is too weak and has low prediction accuracy, the large model will frequently reject the speculative results, triggering regeneration. In this case, speculative decoding not only fails to accelerate inference but actually increases overall latency due to the added overhead of running the small model. An ideal draft model should be a "slimmed-down" version of the target model (e.g., obtained through distillation), remaining as lightweight as possible while maintaining consistency in basic language distribution.

2. Acceptance Criterion

One cannot simply require the tokens output by the large model to match those of the small model exactly (since LLMs are probabilistic). Typically, variants of **$\text{argmax}$** or **Nucleus Sampling** are used. As long as the token selected by the small model has sufficient probability weight in the large model's distribution, it is considered "correct."

3. KV Cache Synchronization

To enable efficient verification, the large model must be able to process the speculative sequence rapidly. This means the KV Cache must support fast append and rollback operations. When speculation fails, the system must quickly roll back the cache to the position of the last accepted token.

Practical Performance and Use Cases

Speculative decoding performs most significantly in the following scenarios:

- **Low-Entropy Text Generation:** Such as code completion, formatted documents, or highly repetitive conversations. In these scenarios, the hit rate of the small model is extremely high, achieving speedups of $2\times \sim 3\times$.

- **High-Latency API Services:** When users are less sensitive to Time to First Token (TTFT) but have requirements for overall throughput, speculative decoding can significantly increase Tokens Per Second (TPS).

However, during deep logical reasoning or creative writing, the unpredictability of tokens increases, causing the acceleration effect of speculative decoding to drop significantly.

Summary

Speculative decoding is essentially a strategy of **trading computation for time**. It acknowledges the reality of redundant computation in LLM inference—we do not need to engage the entire "brain" every time to decide if the next word is "the." By constructing an efficient "fast-slow collaboration" mechanism, AI systems can substantially reduce inference latency without compromising output quality.