Calculate Your Retry Budget: What Does a Single LLM Failure Really Cost?

Last week, a friend’s production service triggered alerts in the middle of the night. Everything looked normal on the model provider’s side, but 23% of our traffic was being retried. The more telling evidence was in the bill: a significant portion of the API costs over the past month came from 429 retries that were essentially repeating calls destined to fail—during that period, the provider was degrading service, and retries could never succeed, yet the money kept burning.

The problem isn’t retrying itself, but rather hardcoding arbitrary constants like “retry three times” without thinking. No one actually calculates: what is the probability of recovering from this failure, and how much are the burned tokens worth?

How you handle 429s, 5xx errors, and timeouts in your LLM pipeline directly determines your billing structure and the system’s true availability. Let’s crunch these numbers today.

The Real Cost of a Single Failure

A single failed retry costs at least two layers of money: the cost of already consumed tokens plus the full prompt that must be resent during the retry.

The biggest difference between LLM calls and traditional HTTP APIs is that money is burned before the failure even occurs. If a timeout happens halfway through output generation, the provider still charges you. Input-side costs—such as system prompts, contexts spanning tens of thousands of tokens, and tool call results—are unrecoverable the moment the request is sent.

Therefore, a successful retry that “saves” the call costs at least 2× the single-call cost. A failed retry costs 3× and yields nothing.

Which Failures Are Worth Retrying

Categorize based on “recovery rate after retry”:

- **429 / 5xx / Network Timeouts**: High recovery rate (429s usually pass after a single backoff). Worth retrying.

- **Output Truncation (`finish_reason=length`)**: Conditional retry—preserve the generated prefix and continue, rather than resending the full prompt.

- **400 Parameter Errors / Content Policy Blocks**: Zero recovery rate. Retrying just burns money for nothing. You must fail fast on the first attempt.

Timeouts are tricky. If the server is still computing but the client gives up first, retrying is equivalent to “burning half the money and gambling again.” The key metric is the ratio between your timeout threshold and the p99 generation duration. If the timeout is far shorter than the p99 latency, “false timeouts” will artificially inflate both your retry rate and your bill.

A ~30-Line Implementation

The core principle is **budget + classification**, not blind looping:

import time, random
RETRY_ON = (RateLimitError, ServerError, TimeoutError)

def call_with_budget(fn, max_retries=2, base=0.5):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except BadRequestError:      # 400 / policy: do not retry
            raise
        except RETRY_ON:
            if attempt == max_retries:
                raise
            time.sleep(base * 2 ** attempt + random.uniform(0, 0.3))

Three deliberate choices here: `max_retries=2` instead of the seemingly “safer” 3 or 5 (explained below); exponential backoff with random jitter to prevent multiple instances from hitting the same rate-limit window simultaneously; and immediate raising of `BadRequest`/Policy errors to bypass the retry loop entirely.

Why 2 Retries, Not 3

Empirical data shows that the recovery rate for 429s after the first backoff is typically above 95%. The second retry adds only about 3–4% additional recovery benefit. Beyond the third attempt, the percentage of calls saved drops sharply, while costs rise linearly.

“More retries mean more safety” sounds right, but if you plot the costs of 3 vs. 2 retries: using 3 retries means spending 15% of your bill to recover <2% of calls. For systems with tens of thousands of monthly calls, this is a pure loss.

Thus, cutting `max_retries` from 3 to 2 typically saves about 10% in retry costs, in exchange for only a ~0.5% drop in overall system availability. Most systems come out ahead with this trade-off.

Two Metrics to Monitor

**Retry Cost Ratio** = Token costs generated by retries / Total costs. A healthy value is <5%. If it stays at 8–15% long-term, it indicates high instability from your dependency, and you should evaluate fallback models or multi-vendor strategies.

**Retry Recovery Rate (by attempt)** = Number of calls succeeded on the Nth retry / Number of initially failed calls. If the recovery rate for the 2nd retry is still above 10%, your failure pattern might warrant one more attempt. If it has dropped below 3%, then `max_retries=2` is the correct boundary.

Monitoring these two metrics together is far more reliable than blindly following the “standard 3 retries” convention.

Summary

Retrying is not a matter of faith; it’s an arithmetic problem. Here are three actions you can implement this afternoon: move 400/policy errors out of the retry path (fail fast on the first attempt); reduce `max_retries` from 3 or 4 to 2, adding comments to explain the rationale; and add a “Retry Cost Ratio” metric to your monitoring dashboard, triggering a review if it exceeds 5%.