Three Lines of Configuration We Missed, Discovered Only a Month After Migrating to Local Inference

Last month, we switched our production inference from a cloud API to a local router. The metrics looked impressive: P95 latency dropped from 1400ms to 220ms, and our end-of-month bill was reduced by roughly 70%. We followed the plan: completed the cutover, observed for a week, and reported the results.

A week later, a user reported that in several recent long conversations, the system had incorrectly quoted something they had said three days prior, attributing it to the wrong context. It wasn’t a hallucination; it was a collision in the retrieval database. It took us three attempts to reproduce the issue, and another day to pinpoint the cause. In the cloud API era, the gateway had a request-log sanitization logic that conveniently standardized the separators in user-defined IDs to underscores. The local router lacked this logic, so IDs were stored as-is. Consequently, two unrelated user IDs in the legacy database collided.

During the migration, we performed full end-to-end regression testing. Our specification even included a clause for “uniqueness validation of custom IDs.” However, we executed these tests using test data—data whose IDs had never collided. As a result, all specs passed with green lights, and we proceeded with the launch.

No one made a mistake per se, but there was a gap in our mechanism: migration tasks lacked a mandatory step to check for hidden side effects in the legacy system.

Our approach has now changed. It’s clumsy, but effective:

First, before any migration, we itemize the configurations, middleware, and routing rules of the previous-generation gateway into a checklist. Each item can only be marked with one of three statuses: Explicitly Migrated, Confirmed Unnecessary, or Unconfirmed. Work cannot begin until the “Unconfirmed” category is cleared to zero. The first version of our checklist had 47 items, nine of which remained “Unconfirmed.” The source of the incident was one of those nine.

Second, test data must not be “clean.” The ID database must intentionally include a set of known colliding IDs. The goal isn’t to verify that retrieval works correctly, but to ensure that alerts are triggered when collisions occur. Previously, our test database was manually crafted and never contained collisions, so we never discovered that the alerting pathway simply didn’t exist.

Third, traffic switching doesn’t happen via an instantaneous flip. Instead, we route a small portion of real traffic through both the old and new systems in parallel. Both sides receive the same requests, and we compare output differences. If the difference exceeds a threshold, the cutover is blocked. During the first parallel run, the discrepancy rate was 0.3%, while our threshold was set at 0.5%, so we proceeded with the launch. Had the threshold been 0.1%, this issue would have been exposed before going live.

The costs are tangible: compiling the checklist takes about two person-days, and writing the comparison scripts takes two nights. We traded this time to avoid a “user-discovered” incident, and no similar issues have recurred in the past three months. The sample size is too small to draw broad conclusions, but for us, it feels more reliable than seeing all specs turn green.