Local Model Access Gateway: Lessons Learned After Three 502 Errors

Last month, we added a local Qwen inference service to our internal knowledge base. The initial integration went smoothly, but problems started appearing during the second round of testing: identical requests would sometimes return in 8 seconds, while other times they would hang for 60 seconds before resulting in a direct 502 error. Upon reviewing the logs, we realized the issue wasn't with the model itself, but with how we integrated it.

**Pitfall #1: Hardcoding direct connection addresses in business code.** Initially, we had three business modules directly connect to the inference container via `http://192.168.x.x:port/v1/chat/completions`. Every time the container restarted or the port changed, we had to update the code in three different places. Later, we introduced a local gateway (a single-port proxy responsible for authentication, model routing, and timeout control), so the business side only needed to recognize one address. A gateway failure is an explicit fault, whereas a direct connection failure is implicit—each business module silently times out on its own, ultimately manifesting as "intermittent feature unavailability," which requires digging through logs from three different projects to troubleshoot.

**Pitfall #2: Setting timeouts too generously.** We initially set the timeout to 120 seconds. During an OOM (Out of Memory) restart, all requests hung for the full 120 seconds before failing, saturating the upstream connection pool and slowing down even features that didn't depend on the model. We reduced the inference call timeout to 15 seconds and configured a fallback route (cloud-based) using the same model. If no response was received within 15 seconds, traffic switched to the fallback. After implementing this switch, the overall P95 latency actually decreased because slow requests no longer occupied threads.

**Pitfall #3: Lacking a minimal verification test case.** During integration testing, we simply verified connectivity with a "Hello" prompt before going live. However, "Hello" takes the shortest path: no tool calls, no long context, and no streaming output. The real issues arose with streaming responses—when the inference container disconnected midway through a long output, the gateway failed to properly close the upstream connection, leaving the client with truncated JSON. We subsequently established a fixed smoke test script with five cases: short queries, long context, streaming, tool calls, and truncation of ultra-long outputs. Every time we modified the gateway configuration, we ran the smoke test before shifting traffic.

**Pitfall #4: Fake health checks.** Our initial "health check" merely verified if the container was running. However, a living container doesn't guarantee the model is usable—weight loading failures, VRAM fragmentation, or stuck backend workers would still show the container status as "Up." We changed this to sending a minimal real inference request (8 tokens) every 30 seconds; two consecutive failures would remove the route from rotation and trigger an alert. The day after deploying these real probes, we caught a "false healthy" state where the container was "Up" but had been unresponsive for 60 seconds.

**Pitfall #5: Retries amplifying failures.** The gateway initially automatically retried timed-out requests three times. When the backend slowed down once, the request volume instantly tripled, crushing the already sluggish backend and expanding the scope of the failure from "occasional timeouts" to "total unavailability." We later tightened the retry strategy: retry only once for explicit 429/503 errors, do not retry on timeouts, and immediately switch to the fallback route instead.

The common lesson: **Inference services are dependencies, not features.** Like databases, they require timeouts, retries, fallbacks, and monitoring, rather than assuming they will always be online. Our gateway now consists of just four components: authentication, routing tables, timeout/failover logic, and real-request probes. At its core, it's simply a proxy process plus a configuration page.

The real complexity lies in recognizing this: the advantages of local inference are cost control and data staying within the intranet, but the trade-off is that it *will* go down, often silently. Designing the access layer by treating it as a fallible dependency saves far more troubleshooting time than obsessing over the model itself.