All Three Languages Published Successfully, but the English Code Block Disappeared: Structural Drift in Translation Output Went Unnoticed

After the 8 PM Sunday run completed, the publishing script logged success for all three languages, and the cover image was correctly linked. The next morning, a reader commented that the `curl` example in the English version was a single block of plain text—no line breaks, no syntax highlighting—making it impossible to copy and run. Checking the original content in the CMS, we found that the ``` fences were completely missing from the English `content_md`. The translation step had dismantled the code block into ordinary paragraphs, merging it into the surrounding context.

Recreating the Scene

Our trilingual pipeline works as follows: zh-CN serves as the source draft; the router calls the model to generate zh-TW and en translations; finally, it pushes all three language versions to the CMS under the same slug. On the day of the incident, the Chinese source contained a YAML configuration snippet followed by two lines of `curl` commands—standard fare for delivery documentation. This seemingly ordinary content exposed two critical weaknesses:

- The translation prompt only specified "preserve Markdown format," with no subsequent validation.

- The publishing script only checked for HTTP 200. A 200 status means "received and written," not "structurally correct."

For the model, merging fenced blocks into paragraphs is a "low-risk modification": the meaning remains unchanged, and the token count stays roughly similar. At a temperature of 0.3, producing such variants is entirely plausible. Worse still, the HTML rendered on the frontend looked readable enough; without a character-by-character comparison against the original, the error went undetected.

The Added Safeguard

We introduced a structural fingerprint comparison before pushing the three languages to the CMS. The fingerprint consists of three lists:

1. The sequence of heading levels (order and count of h1/h2/h3).

2. The number of ``` fenced blocks, along with the first and last lines of each block.

3. The number of tables and the row count for each table.

We extract the baseline from zh-CN, then extract fingerprints from each translated version once, comparing them item by item. If they don't match, we automatically retry the translation once. If it still fails, we block that specific language version and trigger an alert rather than forcing publication. It is better to miss one language and remind humans to fill the gap than to publish a structurally broken version. The implementation is minimal: fingerprint extraction is a simple line-by-line scanning function, and the diff compares three tuples, totaling fewer than 100 lines of code. This safeguard doesn't target rare corner cases but rather addresses the "long-tail slight deviations from the same prompt" phenomenon.

Here is a common pitfall, based on our own experience: Initially, we only compared the "number of fenced blocks." This led to false positives for two consecutive days—once because the translation merged an explanatory sentence outside the fence into the block, and another time because a trailing line inside the block was missing. Comparing counts alone cannot catch "an extra pair of fences" or "block content being moved." Therefore, anchors for each block must include the starting and ending content. Also, do not tighten thresholds immediately: run in parallel for a week first, logging without blocking, to establish rules based on the actual distribution of structural drift before upgrading to hard gates.

Data

Records from ten days of parallel mode operation: 13 translation outputs, 9 passed on the first try, and 4 exhibited structural drift. All drift types fell into the same category: heading counts were correct, but fenced content was merged, or lists were rewritten into prose. After changing the gate from "blocking the entire batch" to "automatic retry once per language + logging diffs," the second-pass success rate reached 100%, with zero human intervention required. The initial pass rate seemed decent, but the failure modes were highly concentrated, indicating that this safeguard addresses real problems, not noise.

Advice for Those on the Same Path

If your pipeline uses LLMs for translation or rewriting, do not accept "the model claims it preserved the format" as your acceptance criterion. You can build a structural diff check in ten minutes: extract three fingerprints (heading sequences, fenced blocks, and tables) from the source and each translation, then compare them item by item. The benefit is that issues are caught before reaching production, and you have full context—which language, which step, and exactly where the diff lies. The cost of content pipelines, like engineering operations, lies not in the difficulty of any single step, but in the fact that every step "looks fine," and "looking fine" is not the same as being verified.