Engineering Practice: Circuit Breaking and Degradation in AI Inference Pipelines

In our daily lab experiments at SFD, we maintain a local AI inference stack. As mentioned in a previous article, routing all production inference through external calls can occasionally lead to issues such as unavailable routes, high queuing latency, or even complete pipeline failures. This article adds a critical engineering perspective: when external routes are unavailable, can your local model automatically take over?

Problem Scenario

Assume you are running a complete local inference stack with the following configuration:

ROUTER=http://127.0.0.1:4000
# Upstream model
UPSTREAM=qwen3.6-plus
# Local fallback model
LOCAL_MODEL=Qwen3-Embedding-8B

When you call `qwen3.6-plus` via the router, the router attempts to forward the request to the upstream service. If the upstream is unavailable (due to network timeouts, service downtime, or expired API keys), the router typically returns a 502 or 504 error.

**The Problem**: In this scenario, your local model is not triggered, causing the entire request chain to break abruptly.

Engineering Solution: Circuit Breaking + Degradation

The core idea of this solution is to **configure a fallback strategy at the router level, automatically switching to the local model when the upstream model returns specific error status codes.**

Configuration Example

In `~/.openai-router/config.json`, you can define `fallback_on_status` for each model:

{
  "models": {
    "production-gpt": {
      "upstream_id": "gpt-4",
      "deployments": [{
        "api_base": "https://api.example.com",
        "api_key": "${API_KEY}"
      }],
      "fallback_on_status": [408, 429, 500, 502, 503, 504],
      "fallback_model": "local-gpt-3"
    },
    "local-gpt-3": {
      "deployments": [{
        "api_base": "http://127.0.0.1:8050",
        "model": "gpt-3.5-turbo"
      }],
      "timeout_sec": 120
    }
  }
}

The `fallback_on_status` field defines which HTTP status codes trigger degradation. When the upstream returns errors like 502 or 504, the request is automatically routed to the `local-gpt-3` model.

Real-World Case

On one occasion, our upstream inference service underwent maintenance early on a weekend morning. Requests sent to the upstream timed out while waiting, ultimately resulting in a 504 error. Since we had configured `fallback_on_status: [504]` and specified `fallback_model: "local-model"`, the requests automatically degraded to the local model. Although the local model's inference speed was slower (30 seconds vs. 5 seconds), users did not perceive any service interruption.

Key Considerations

1. **Response Quality of the Degraded Model**: Local models often differ in precision from upstream models, potentially yielding different results after degradation. For critical business operations, evaluate whether the degraded output meets expectations.

2. **Timeout Settings**: Degraded models are typically slower, so longer timeout values should be set to prevent requests from being terminated prematurely.

3. **Fallback Chains**: Multi-level fallbacks can be configured, such as `Upstream Model → Local Model A → Local Model B`, creating a more robust resilience strategy.

4. **Monitoring and Alerts**: Even with automatic degradation configured, you should monitor and alert on degradation events to understand their frequency and causes.

Summary

Relying entirely on external inference services in production environments carries risks. By properly configuring circuit breaking and degradation strategies in your router, you can maintain basic inference capabilities while ensuring core availability. This approach is not intended to replace external services but serves as a safety net for sudden service unavailability scenarios.

**A reminder to myself**: If you already have a local inference stack, check your router's fallback configuration tonight. Many "service outage" issues can actually be resolved with simple degradation strategies.