Pin Models and Prompts to Specific Versions: Don’t Let Upstream Silently Swap Engines and Capsize Your Tasks at Midnight

At 3 AM, after an automated job had been running all night, the parsing success rate on the dashboard dropped from 99% to 71%. My first instinct was that something was wrong with the data source. It took two hours of digging through logs to discover the embarrassing truth: a few days earlier, the router had performed an upstream pool migration. The alias we had been using, `qwen-cloud-plus`, had been silently redirected to a different model. Not a single character in the prompt had changed, but the underlying weights had. Outputs that previously reliably returned JSON started occasionally returning free-form text, causing a cascade of futile retries in our downstream `json.loads` calls.

Fixing it took just one line of code: replacing the alias with a versioned name containing a date stamp. But this incident made me realize that we had been treating “the model” as a fixed service endpoint, when in reality, it drifts. What lies behind an alias is determined by the upstream provider, and they rarely notify you when they swap out the engine. You assume you are evaluating, regressing, and relying on the same model, but yesterday’s model and today’s might be completely different entities.

Since then, we have pinned three critical elements into our configuration, hardcoding them in our repository so that any change leaves an audit trail.

First is the **model version**. Bare aliases are prohibited in production jobs. Instead, we must use snapshot-style names like `qwen-cloud-plus-2026-08`. Upgrading isn’t as simple as tweaking a string; it requires a comparative review. We run a fixed set of inputs against both the old and new versions, manually inspect the differences, and only advance the configuration pointer once the new version passes review. If the upstream provider swaps pools again, it merely triggers a “new version pending review” status, rather than letting live tasks spontaneously mutate into a different species.

Second is the **prompt itself**. A prompt may look like just a few lines of text, but it is a core asset—and also the element most susceptible to being casually “optimized” and inadvertently broken. Now, every production prompt has its own file and version number. Changing a prompt is treated like changing code: it requires a diff and a peer review. No more letting prompts live solely as string constants in some script or in someone’s head.

Third, and in my opinion the most valuable addition, is a small set of fixed inputs we call the **golden set**. It consists of just ten to twenty items—requests that historically caused issues or were used to verify correct outputs, paired with their “standard answers.” Whenever we upgrade a model, modify a prompt, or undergo a passive upstream migration, we run the entire golden set and verify each item individually. This is your only regression metric that doesn’t depend on live traffic. Live samples drift and carry business noise, but the golden set is a baseline you have explicitly validated.

Some might ask whether maintaining a dedicated benchmark for just ten or twenty items is worth the effort. My answer is that those two percentage points in parsing success rate represent task queues backed up until 9 AM and half a day of manual cleanup. The effort invested in the golden set transforms “midnight capsizing” from a probable event into a near-impossibility.

Finally, here’s a lesson learned the hard way: pinning versions isn’t enough; you must also expose them explicitly. Now, the first line of the startup log for every job run clearly states the model version, prompt version, and build time. When issues arise, there’s no need to guess which stack was in use that day—a single log line provides definitive confirmation. Previously, we invested in observability tools like heartbeats and alerts to answer “Is the task running?” These three pinned-version practices answer “Is it running the stack I think it is?” These are distinct concerns, and both are essential.

To wrap up: Treat models as drifting dependencies, much like managing a third-party library whose release cadence you don’t control. Aliases are for human convenience; versions are for production reliability; and the golden set is the ruler you use to judge whether the swapped-in version actually works.