Depart Only When Requests Are Uneven: Continuous Batching in Inference Services

Last week, a reader asked: "Why do others handle three times the request volume on the same GPU? The model, quantization, and hardware are identical." The gap often lies not in the model, but in scheduling.

Why Batch in the First Place?

For every forward pass during inference, the entire model's weights must be read from VRAM. A 70B parameter model weighs approximately 140GB (in fp16). Processing a single token for one request means moving this 140GB of data just to produce one token, leaving compute power waiting on memory bandwidth. If two requests are processed together, the weights are still loaded only once, doubling the output. This is the first principle of batching: weight reading is a fixed cost, while tokens are marginal output; the larger the batch, the thinner the amortized cost.

The trade-off is latency for throughput: the response time for a single request gradually increases with the number of requests in the same batch. Therefore, the core question of batching strategy is not "how large should the batch be," but "which requests should be computed together."

The Achilles' Heel of Static Batching

Early inference services commonly used static batching: accumulate N requests (or wait until a timeout) and send them to the GPU together. After this batch is completed, the next batch is accumulated.

The problem with "sending together" is that it implies "graduating together." Suppose a batch contains four requests: the shortest generates 50 tokens, while the longest requires 800. The shortest request could have returned after 50 steps, but it must wait for the entire batch to finish the 800 steps. The GPU continues computing the 800th step for the three faster requests, producing tokens they don't actually need (each requires only 1 step per token), wasting compute power.

Furthermore, the rule of "accumulating N requests" is rigid when new requests arrive. Even if a new request could join within 20 milliseconds, it must stand by the roadside waiting for the current "bus" to finish its trip, which often takes hundreds of milliseconds to several seconds.

Combined, these factors result in the typical GPU utilization of static batching hovering between 10% and 30%.

Continuous Batching: Swapping Passengers Step-by-Step

Continuous batching (also known as in-flight batching) downgrades the concept of a "batch" from the request queue level to the token step level. Before each forward pass, the scheduler checks: Which requests generated an EOS (End-of-Sequence) token in the previous step? Kick them out of the current batch and immediately reclaim their KV cache VRAM. Are there new requests in the queue? Slot them into this batch, sharing the operator launch overhead of the same forward pass.

The effects are threefold:

1. **Short requests no longer ride along unnecessarily.** A 50-token request returns after 50 steps, freeing up VRAM and compute for the remaining 750 steps for new requests.

2. **New requests don't wait for the bus.** Initial latency drops from "waiting for the entire batch interval" to "waiting until the current step ends," typically in the range of tens of milliseconds. Users perceive faster service, reduced queuing, and a significant drop in P99 latency.

3. **Batch size becomes naturally elastic.** What the GPU computes at any given moment is effectively the "current batch," which might consist of 64 requests that change with every step. Utilization follows the load, rather than relying on the luck of timed polling.

The cost is increased complexity in memory management: KV cache must be allocated and released at the request granularity, and the scheduler must maintain a "boarding list" for each step. This is precisely what vLLM's PagedAttention aims to solve—splitting the KV cache into fixed-size blocks and managing them like operating system virtual memory pages, thereby eliminating fragmentation and pre-allocation issues.

Practical Recommendations

- Check your running engine and scheduling mode. TF-Server defaults to static batching; vLLM, SGLang, and TensorRT-LLM default to continuous batching. If your P99 tail latency is high and GPU utilization remains low, suspect the scheduler before blaming the model.

- Don't treat batch size as a dogmatic parameter. Under continuous batching, it is more important to monitor `max_num_seqs` and the total KV cache capacity: the former limits the number of concurrent requests, while the latter limits the total amount of KV data that can fit into VRAM simultaneously. If concurrency cannot increase, it is usually because KV VRAM has hit its limit. Adding more request slots is useless; instead, either reduce the context length limit or use quantized KV cache.

- When mixing long and short requests, set a `max_tokens` upper limit for long requests. An uncontrolled 4K-token long response can bog down the overall turnover of a batch. Setting a limit is a protective measure, not a restriction on quality.

In short: No matter how fast a single forward pass is, it cannot save requests stuck waiting for the next bus. It is far more cost-effective to keep the GPU fully occupied with useful work every 20 milliseconds than to have it run full batches only a few times an hour.