Adding Rate Limiting to Large Language Models: Why Higher QPS Limits Aren't Safer

Many teams treat "QPS limits" as a large number that they gradually decrease until business alerts trigger, at which point they lower it slightly more. However, rate limiting at the gateway plays a more subtle role: it is the only buffer between you and your model inference cluster. Today’s topic is how to configure this buffer, and many people’s initial intuition about the answer is wrong.

Let’s start with the conclusion: **The correct limit value depends on your cost-of-failure structure, not on estimated peak traffic.**

What Gateway Rate Limiting Actually Prevents

Model services are the most expensive downstream component in such systems: a single inference request can occupy GPU memory for tens of seconds to several minutes. Request buildup directly degrades TTFT (Time to First Token). If the gateway does not enforce rate limits, sudden traffic spikes will quickly fill up the GPU queue—not causing a service crash, but causing response times for all requests to tail off, resulting in a user experience where "the entire site slows down."

Therefore, the true goal of rate limiting is not to prevent overload, but to **transform overload into predictable queuing behavior**. This is quite different from traditional API rate limiting.

Three Common Rate-Limiting Approaches and Their Pitfalls

**Fixed QPS Cap**: The simplest approach, but the problem is that you are using a constant to constrain a downstream system whose distribution changes drastically over time. A QPS of 200 during morning peak hours and only 15 at night means that the same cap of 100 results in completely different queue lengths in these two periods.

**Window Counter (Allow N requests every 10 seconds)**: Slightly more stable than raw QPS, but it suffers from severe boundary effects—at the very beginning of the 10th second, up to 2N requests might be allowed instantly between two windows. Model services are extremely sensitive to second-level bursts, and these spikes are exactly what you want to avoid.

**Token Bucket**: Maintains a bucket that fills with tokens at a constant rate, with each request consuming one token. Compared to the window counter, it naturally allows moderate bursts (when there are surplus tokens in the bucket) but prevents unlimited burst volumes. Most production gateways use this method.

How to Determine Parameters: Reverse-Engineering from Inference Latency

There are two parameters on the gateway side that must be tuned:

- **Fill Rate (rps)**: The number of tokens added per unit of time.

- **Bucket Capacity (burst)**: The number of requests that can be passed through at once during a burst.

The correct approach is to reverse-engineer from the downstream system rather than guessing. The data you need includes:

- The p95 latency of model inference (not the average; tail latency determines queuing behavior)

- The maximum concurrent processing capacity of the backend GPU pool

A rough but practical derivation: If your concurrency limit is C and the average latency is T seconds, then the rps the system can stably handle is approximately C / T. Set the gateway rps to 0.7–0.8 times this value to leave room for buffering.

**Bucket Capacity** involves another trade-off. If it’s too large, burst spikes can still hit the downstream system; if it’s too small, normal fluctuations will be mistakenly blocked. A good empirical starting point is 1.2–1.5 times C—enough to accommodate reasonable irregular traffic without turning an under-saturated state into actual backlog.

The Counterintuitive Part

Many people’s first reaction after setting limits is: "If the rejection rate (percentage of rejected requests) is 0%, it means my limits aren’t strict enough." This is the most common misconception.

**A zero rejection rate usually indicates that you have a large amount of idle capacity wasting money.** Moreover, in a system that rarely triggers rate limiting, you have no idea how it behaves when limits are actually pressured—the next time traffic doubles, you’ll encounter the cliff edge for the first time in production.

A more robust approach is to intentionally send a load 50% higher than your estimated peak in a staging environment, observe where the TTFT degradation curve begins to steepen, and use the QPS near that inflection point as your baseline for upper and lower limits.

A Checklist

Before publishing rate-limiting rules, go through these three items:

1. Are your rps and bucket capacity derived from p95 latency, or just a number guessed by operations?

2. What is the behavior when rate limiting triggers—a direct 429 error, or queuing with a timeout? (Direct 429 errors usually have a much higher false-positive rate than queuing + waiting.)

3. Do you have a way to see "how much headroom remains before the next 429" without writing code? If you can’t see this metric, the next scaling event will be a reactive response rather than a proactive one.

Rate limiting is not a mechanism to punish users; it is a tool to make costs predictable. Setting limits based on reverse-engineered values and calibrating them with observational data is much safer than the approach of "setting it high initially and lowering it after issues arise."