The Age of Inference Acceleration: Why Speculative Decoding Can Make LLMs 3x Faster

When running inference tasks at SFD Lab, we spent a frustrating period watching identical 70B models perform at 40 tokens/s in some configurations and barely 15 tokens/s in others — nearly a 3x difference. The culprit turned out not to be the hardware, but whether Speculative Decoding was enabled.

This article takes a serious look at the technology — not the one-liner explainer level, but a genuine breakdown of the underlying mechanics, combined with our real-world performance observations.

Why Is LLM Inference So Slow?

Before understanding Speculative Decoding, we need to understand why LLM inference is slow in the first place.

Large language models are autoregressive — they generate one token at a time, with each new token depending on all previous context. This process cannot be parallelized; it must proceed step by step. The compute is dominated by matrix multiplications, and a full forward pass is required for every single token.

This is the memory bandwidth bottleneck. For large models, the primary inference cost is not computation but the process of loading model parameters from GPU memory into compute cores. More parameters means slower per-token forward passes.

The Core Idea of Speculative Decoding

Speculative Decoding has an intuitive premise: use a small model to guess several tokens ahead, then let the large model verify the entire batch in a single pass.

1. Draft phase: A much smaller model generates k tokens consecutively. This is fast because the small model has low inference overhead.
2. Verify phase: Feed all k tokens to the large model for a single parallel forward pass.
3. Accept/Reject: Check each position left to right. Accept where the small model matches the large model distribution. Truncate at the first mismatch.

When the acceptance rate is high (typically 70-85%), you effectively get several tokens for free.

Real-World Performance

We tested on a Mac Studio M3 Ultra (96GB) at SFD Lab:

• Qwen2.5-72B + Qwen2.5-7B draft: Average speedup ~2.4x. Code generation dropped to ~1.8x due to lower acceptance rates.
• Llama-3.1-70B + Llama-3.1-8B: Consistent 2.8-3.2x speedup. Same-family models have closer token distributions.
• DeepSeek-R1 671B: RPC distributed setup still in progress.

Is There Quality Loss?

Theoretically, Speculative Decoding is strictly equivalent — the Modified Rejection Sampling mechanism ensures the final output distribution is mathematically identical to pure large model generation. In our tests, Perplexity scores showed almost no change.

Implementation Considerations

• Draft and target models should come from the same family
• Memory pressure roughly doubles — a 7B + 70B combination at 4-bit needs ~45GB VRAM
• Major frameworks support it: vLLM, TGI, llama.cpp (--draft-model), Ollama (select versions)

Summary

Speculative Decoding delivers 2-3x speedup with no quality trade-off. The cost is extra VRAM and a compatible draft model. For teams running LLMs locally, this technology merits serious evaluation.