How to Break Down LLM Costs: Understanding Model Cascading

When many teams discuss cost reduction, their immediate reaction is often to switch to smaller models or simply cut budgets. Both approaches are rather blunt. A more controllable alternative is called **Model Cascading**: let cheaper models handle requests first, and only hand off to more expensive models when the cheaper ones are "uncertain."

The Principle

Take intent classification as an example. When a customer service message arrives, it goes through a two-stage pipeline:

1. A small model performs initial screening to determine whether the message is an inquiry, a bug report, a complaint, or invalid noise.

2. If the confidence score provided by the small model exceeds a preset threshold (e.g., 0.9), the result is accepted directly. This request incurs only the token cost of the cheap model.

3. If the confidence is below the threshold, the request is escalated to a stronger model for re-evaluation. The small model’s initial judgment is included in the context for reference.

The math is straightforward: let $C_1$ be the unit cost of the expensive model and $C_2$ be that of the cheap one. If the escalation rate is $p$, then the average cost per request is $C_2 + p \times C_1$. If 80% of requests are handled by the cheap model (an escalation rate of 20%), the overall cost drops to slightly above 40% of the original. Loosening the threshold to reduce the escalation rate to 10% lowers costs further, albeit at the expense of increased quality risk.

The Real Workload: Data and Thresholds

Implementing cascading isn’t just about setting up the pipeline; there are three engineering tasks you can’t skip.

**Define escalation conditions.** Model-reported confidence scores are the most intuitive metric, but many inference frameworks either don’t output logits or require temperature-based renormalization to make them usable. A fallback approach is to use heuristics: if sampling twice yields different answers for the same question, consider the model uncertain. Outputs that are too short, miss required fields, or violate format constraints should also trigger escalation. These rules are more stable than raw confidence scores, which can drift with minor prompt tweaks.

**Build a golden dataset.** Extract 500–1,000 historical requests, have the strong model generate reference answers, and then run the cheap model on the same set. What you’re really tuning is the threshold curve: determining at what escalation rate the drop in accuracy remains within an acceptable range (e.g., under 1%). Without this dataset, your threshold is just a guess, leaving you to rely on live error logs after deployment.

**Provide a safety net for the expensive model.** What happens when the escalation chain is fully loaded, or the strong model times out or errors out? Do you return the small model’s result, or explicitly state "unable to judge"? While fallbacks might work in customer service scenarios, silently substituting cheap model answers for expensive ones in compliance-sensitive or financial contexts is a serious incident. This strategy must be finalized before traffic shifting and integrated into alerting systems.

When Not to Use It

There are three scenarios where cascading actually leads to losses:

1. **Low upstream traffic:** If you receive fewer than a few hundred requests per day, the person-days spent tuning the system will outweigh the savings on tokens.

2. **Strongly sequential tasks:** For tasks like long-document translation, where the cheap model produces a draft and the expensive model still needs to review the entire text, the total cost of "cheap first, then expensive" may exceed simply calling the expensive model directly.

3. **No measured escalation rate:** If you design the pipeline based purely on guesses without testing, first instrument your logging to measure the actual distribution before proceeding.

The Minimal Implementation Path

If you want to try it, four steps are sufficient. Choose a task with over 1,000 daily requests and clear correctness criteria, and build a golden dataset of 500 samples.

1. **Start in shadow mode:** Run the full cascading pipeline internally, but continue returning the strong model’s results to users. Only log the differences and latency between the two paths.

2. **Align for one to two weeks:** Once consistency rates and latency meet standards, begin shifting traffic. Start with 20% (one-fifth); do not shift all traffic at once.

3. **Monitor the escalation rate:** After shifting, keep a close eye on this single metric. A sudden spike often indicates data drift, prompt changes, or upstream anomalies. This signal appears earlier than errors in logs.

Model cascading isn’t mysterious. At its core, it shifts the paradigm from "always use the most expensive option" to "pay according to uncertainty." The challenge lies in data and thresholds, not architecture.