How Renting a Local Inference Cluster Led to "False Health": Three Pitfalls Encountered During Downtime Drills

Last week, during the final stretch of integrating our local inference cluster into production, everything looked perfect: CI was all green, monitoring was all green, and the fallback链路 (fallback path) drill even appeared "successful." However, on the first day we actually blocked upstream traffic, we got stuck at the very first step of the recovery process. Following our incident response plan, we performed a rolling restart of "unhealthy nodes," only to find that the node we took offline was actually the healthiest one.

The problem wasn't with the restart script; it was that we had defined two different standards for "health." Both standards worked fine under normal conditions, but they conflicted with each other during an incident.

Pitfall 1: Two Health Checks, Two Truths

At the time, we had two layers of health judgment:

- **Probe Layer**: Periodically hit `/health`; if it returns 200, the node is considered alive.

- **Load Layer**: Mark nodes as "overloaded" based on queue depth and queuing latency.

During the downtime drill, the router removed overloaded nodes from the scheduling pool. When we reviewed the logs the next day, we found that the removed nodes were all green from the probe layer's perspective but red from the load layer's perspective. Neither side was wrong, but our recovery playbook stated, "Restart nodes with failed probes." Following this instruction, we ended up restarting the node with the highest load—which was still technically alive.

We later hardcoded this distinction into our emergency response plan: **Use load-layer metrics for scheduling removal, and use process-level probes for restart actions. The label "unhealthy" is not shared between these two mechanisms.** Now, this comparison table is posted on the first page of our recovery manual. It might look like just documentation work, but it’s really about decoupling the two judgment layers, saving us from one mistaken restart at 4 AM.

Pitfall 2: The Fallback Drill Was "Successful" Because There Was No Real Traffic

Our router configuration was: primary route to `127.0.0.1:4000`, with fallback to the cloud. The drill method involved killing the primary router process and sending a few test requests—all of them went through the fallback, so the drill passed.

However, during the first real production incident, we discovered that the timed-out requests never even reached the fallback decision point. They got stuck at the connection layer between the client and the primary router. The TCP connection was established successfully, the write succeeded, but the read timed out. The client’s retry logic immediately retried three times in place. The fallback mechanism only kicks in when "a request successfully reaches the router and is rejected."

The fix we implemented: reduced the client-side timeout from 30 seconds to 8 seconds and added an idle timeout for the connection pool. On the router side, we added a rule to "actively return 503 when the queue exceeds a threshold," ensuring that requests fail before reaching the fallback decision point, rather than hanging in the queue. The lesson from this drill wasn’t that the router was misconfigured, but that **the failure points for drill traffic and production traffic were at different layers**, and we had only validated the layer covered by the drill.

Pitfall 3: Using the Same IP for Both Router and Asset Source

Another detail: our local router process also hosted the HTTP endpoint for static cover images. When the router reloaded its configuration, the task fetching new article cover images was interrupted simultaneously, causing the article publishing workflow to report "Cover Image 404." It took nearly forty minutes of troubleshooting to realize that both services were running on the same process and port.

The solution was simple: move static files to an independent port, so the router’s image endpoint handles only images. One hour of downtime bought us a new rule: **One port should handle only one failure domain.** Anything else attached "incidentally" must have its own separate incident response plan.

Wrap-up

Individually, none of these issues are complex. Their commonality is that health determination, failure paths, and failure domains mask each other when the "system is alive," only revealing themselves when the system fails. The value of drills isn’t to get green checkmarks, but to clearly define these three sets of standards and confirm that they describe the same incident scenario.

Now, for every downtime drill, the first item on the checklist isn’t "Service restored," but "Are we using the same set of health labels?"