The "Dynamic Dispatcher" of Modern AI Systems: A Deep Dive into Continuous Batching

In production environments for Large Language Models (LLMs), one of the most significant components of inference cost is GPU utilization. If you observe a simple inference request, you will find that during the generation of each token, the GPU spends most of its time waiting for memory transfers (Memory Bound) rather than performing computations. While traditional Static Batching can improve throughput, it introduces a fatal flaw: the Bucket Effect.

The Pain Point of Static Batching: Waiting for the Slowest

In traditional static batching, the system packs multiple requests into a single batch and feeds them into the model simultaneously. However, the number of tokens generated by different requests varies drastically—some requests may only require a "Yes" response, while others need to generate a long essay.

In this mode, the entire batch must wait until the longest sequence is fully generated (or hits a stop token) before resources are released. This means that after short requests are completed, the VRAM they occupy remains locked, and the GPU must fill in "padding tokens" for these completed requests in subsequent iterations. This waste not only reduces throughput but also incurs unnecessary computational overhead.

Continuous Batching: Breaking the Chains of Synchronization

To address this issue, modern inference frameworks like vLLM have introduced Continuous Batching. Its core logic refines the granularity of batching from the "request level" down to the "token level."

How It Works: Dynamic Insertion and Exit

Instead of waiting for the entire batch to complete, Continuous Batching checks for completed requests immediately after each iteration step.

1. Immediate Exit: As soon as a sequence generates an <EOS> token or reaches its maximum length, it is immediately removed from the current batch, and the result is returned to the user.
2. Immediate Insertion: Within the same iteration step, the system checks the waiting queue. If new requests arrive and VRAM allows, it will be directly inserted into the current running batch.

This ensures that GPU compute units are always processing valid token generation tasks, without wasting resources on padding to align lengths.

Core Support: PagedAttention and Memory Management

The feasibility of Continuous Batching relies on refined management of the KV Cache. If traditional contiguous memory allocation were used, frequent insertion and deletion of requests would lead to severe external fragmentation.

PagedAttention borrows the paging mechanism from operating system virtual memory:
- It splits the KV Cache into fixed-size "Pages."
- The KV Cache for each request no longer requires physical continuity; instead, it is distributed across non-contiguous physical blocks via a mapping table (Block Table).
- When Continuous Batching needs to insert a new request, it simply allocates a few free pages without needing to move existing data.

Practical Impact: Order-of-Magnitude Throughput Gains

From an engineering perspective, Continuous Batching delivers significant performance leaps:
- Increased Throughput: Compared to static batching, throughput typically increases by 2–4 times.
- Reduced Time to First Token (TTFT): New requests can enter the computation stream without waiting for the previous batch to finish entirely.
- Maximized Resource Utilization: GPU computing power is distributed more evenly across all active requests.

Conclusion

If the KV Cache is the "short-term memory" of AI, then Continuous Batching is the "efficient dispatcher" of this memory system. By breaking the deadlock of synchronous waiting, it transforms LLM inference from a "queuing" mode into an "assembly line" mode. For any AI application pursuing high concurrency and low latency, this has become a standard infrastructure-level configuration.