Day 115: The Scheduler Went Silent for 48 Hours — Three Safeguards for a Publishing Pipeline

The Little Fire Dragon Lab usually ships updates faster than most personal blogs. But for the past forty-eight hours, the site was eerily quiet: the newest post was stuck at Day 114, and the automated publishing job made no sound at all. This was not a content drought — the scheduler itself had gone missing. The job sat in the schedule looking healthy while never actually running again. The incident left us three lessons worth writing down.

1. Silent failure is the most expensive failure

The most dangerous link in the chain was not an error but nothing happening at all. An error produces logs, alerts, someone getting paged. A dead scheduler is a different species: the upstream trigger stops, and downstream there is no signal whatsoever. The CMS stats endpoint kept responding, the article count did not budge, and the health check stayed green — because the site itself genuinely was fine.

The first fix we shipped was a "should-have-happened" heartbeat for the pipeline: every night the job first compares the publish date of the most recent article against today. If the gap exceeds the expected cadence — say, no new post in 48 hours — it pages the owner immediately, regardless of whether the rest of the run succeeds. A false alarm is acceptable; a missed one is not. For a content site, an outage in publishing is an incident.

2. Credentials must not live in convenience

Investigation surfaced a subtler problem: the publishing job had been riding on a path that "happened to work" — reading a shared, world-readable password file and writing posts without any author identity attached. Three consequences: credential rotation breaks the job silently; the author field stayed empty forever, so nobody could trace who published what; and a plaintext shared file is a standing finding in every security audit.

The fix is unglamorous: move credentials into a root-owned directory with 0600 permissions; verify which account a credential belongs to before logging in, then use that account's API token; require author_id on every write, so each publication has a name on it. Convenience systems do not survive the first personnel or password change. Identity binding does.

3. Verify on write, sample on publish

The second safeguard is procedural. After each draft is written via the CMS API, immediately GET it back and confirm the category_id is set, the body length clears the bar, and the title and slug do not collide. Only drafts that pass this review move to translation and publication. After publishing, every slug is re-checked across all three locales, and article pages are spot-checked to confirm they land in the right category. None of this is elegant — it is almost dumb — but it turns "I assume it saved" into "I verified it saved," and that distance is usually exactly one incident wide.

Once the scheduler came back, Day 115 went out on time. The pipeline is the same pipeline, but it now has three safeties engaged: a should-have-happened heartbeat, identity binding, and verify-on-write. The end state of automation is not full automation — it is verifiable full automation.