Benchmark Drift: Why Our Weekly Model Rankings Quietly Shift

Last month, we stared at evaluation results that had been running for two weeks and discovered an awkward fact: with the same set of models, Model A ranked above Model B last week, but this week Model B had overtaken it. The score difference was a mere 1.7 points. Engineers debated for an entire afternoon, yet no one could convince the other.

We spent three days investigating and finally confirmed the culprit: it wasn’t the models that had changed, but the benchmark dataset.

Where the Problem Lay

Our evaluation process worked as follows: customer scenarios were anonymized and organized into question-answer pairs, with a new engineer manually adding fresh cases each week. This seemed reasonable—the benchmark should evolve alongside our business needs.

But that was precisely the problem. "Evolving with the business" meant the distribution of the benchmark shifted every week: cases added on Mondays leaned toward consultation-style queries, Wednesdays toward operational tasks, and Fridays toward long-form text generation. The scoring mechanism itself hadn’t changed; what changed was the "coordinate system" for those scores.

One particularly subtle case stood out: for three consecutive weeks, we observed a specific model’s "long-form quality declining." Upon investigation, we found that the new cases added each week happened to include long-text tasks exceeding 2,000 words, while a large number of short Q&A pairs from previous weeks had been archived. Short Q&A was that model’s strength; once these were removed from the benchmark, its average score naturally dropped. The model itself had not changed at all.

Another, even more隐蔽 (hidden) source of drift came from anonymization. When anonymizing customer cases, we replaced specific product names and internal system names with placeholders. For the same business scenario, one engineer might anonymize it as "a certain CRM system" in the first week, while another engineer might write "Customer Relationship Management System" in the third week. Although the surface-level case changed, the evaluation matching logic shifted along with it. We later fixed the anonymization vocabulary into a controlled document, hard-coded the replacement rules, and prohibited any ad-hoc improvisation.

What We Changed

**First, stratified the benchmark with fixed weights.** We divided evaluation cases into five layers based on task type: short Q&A, multi-turn dialogue, long-form generation, code assistance, and structured extraction. We set fixed proportions for each layer—for example, 30% for short Q&A and 25% for long-form generation. When adding new cases weekly, we only added them to their corresponding layers without altering the proportional balance. Thus, even as the total number of cases grew, the distribution remained stable.

**Second, new cases must be labeled with their target layer.** Engineers adding new cases must clearly specify the type upon submission, rather than dumping them into a single large pool. This rule is lightweight but blocks most entry points for drift.

**Third, maintained an immutable "anchor set."** We froze 100 cases extracted from the benchmark, adopting a policy of addition-only—no deletions or modifications. During each evaluation run, we reported scores for the anchor set separately from the full dataset. If the anchor set scores remained stable while the full dataset scores fluctuated, we could confidently attribute the variance to distribution shifts introduced by new cases, rather than model degradation.

**Fourth, score comparisons must include confidence intervals, not just means.** Previously, we looked at statements like "82.3 vs. 84.0, a 1.7-point improvement." Now, we report scores by layer and require each layer to have at least 30 cases; layers with insufficient data are explicitly marked as "insufficient sample size, no conclusion drawn." Regarding that contentious 1.7-point difference, had we broken it down by layer at the time, we would have seen the discrepancy concentrated in the long-form layer, which had only 12 cases—meaning no conclusion should have been drawn in the first place.

An Easily Overlooked Pitfall

We stumbled into a pitfall during stratification: case "types" should be defined by task intent, not input length. A 50-character query requesting a complete SQL output belongs to the structured extraction layer, not the short Q&A layer. Initially, we stratified by input character count, causing SQL-related cases to scatter across three different layers, which completely distorted the stratified statistics. The issue was resolved only after we switched to stratifying by labeled intent.

Summary

A benchmark is not a database where you simply add data without auditing. Its distribution serves as the coordinate system for your models; if the coordinates drift, the rankings lose their meaning. The combination of stratification, an anchor set, and minimum sample sizes required roughly one person-day of effort, but it transformed our workflow from "debating scores every week" to "intervening only when scores show anomalies."

Two weeks later, the fluctuation in anchor set scores was compressed to within 0.4 points, and the week-over-week volatility of the full dataset scores dropped from an average of 2 points to 0.8 points. The previous debates over "Is it A or B?" now only arise when stratified scores truly breach thresholds—and even then, a quick review of the cases usually yields an answer within ten minutes.