SEARCH THE LAB

What are you looking for?

ScienceReview information not provided

Backfilling Nine Days: A Controlled Historical Replay

Review information not provided · 1 min

For two weeks the daily publishing pipeline sat idle. The cause was neither credentials nor a CMS fault: every 02:00 run looked for a content pack that did n…

Backfilling Nine Days: A Controlled Historical Replay

For two weeks the daily publishing pipeline sat idle. The cause was neither credentials nor a CMS fault: every 02:00 run looked for a content pack that did not exist. The log said content_pack_missing and named the exact path. A missing pack means no writes, which is the intended fail-closed behaviour. What went wrong was the reporting: that fact was relayed as a pending CMS username, so nobody refilled the content for twelve days.

Facts

  • Replay window: 2026-11-20, 00:59 to 01:10 SGT, one slot at a time, never concurrent.
  • Scope: daily slots from 11-06 to 11-19 (skills on odd days, science on even days), the monthly article for 11-15, and part of the diary series.
  • Outcome: the ledger now holds 24 slot records, 23 published and one skipped (the 11-16 diary, which had no genuine record for that day).
  • Public checks: /skills/, /zh-tw/ and /en/ all return 200.

Method

The replay deliberately bypassed runDaily, which refuses to write a planned business date earlier than the current Singapore date. Instead each slot was executed against an explicit specification: identity stays the pair of business date and category, such as 2026-11-13|c4. Every article still had to pass the same validation (body field, three locales, matching category, legal cover path), and each write was read back afterwards. Re-running a finished slot is skipped by idempotency rather than duplicated.

What Is Still Missing

Ten diary slots remain unpublished: 11-05, 11-07 through 11-09, and 11-11 through 11-16. Diary evidence rules are stricter: the source file must be named for its date, must carry that date in its own body, and must meet a minimum size. Missing evidence means a skip, never invention. Filling these requires turning each day's real record into a proper source file first.

Takeaway

The pipeline was never broken. The reporting of a missing pack was. Content needs a human; preventing a repeat needs an alerting contract that relays the runner's real stage instead of a fixed sentence.