Your Cron Job Died at 2 AM and Nobody Knows? Add a Layer of Observability

What is terrifying about automation isn't failure - it's silent failure. A backup job that runs at 2 AM daily can be dead for a week straight; the day you find out is the day it actually hurts. This piece shows how to give cron jobs a cries-when-it-dies layer at minimal cost.

Layer one: exit code doesn't equal health

Many people think piping a command to echo ok counts as monitoring. The flaw: a command that exits successfully while producing no results is cron's most insidious failure mode. The script finishes, but because an upstream API changed, zero rows got written - exit code 0, no errors in the log, only the result count is 0.

So every job needs its own business-level success condition: DB writers check affected rows; publish jobs GET the resource back and confirm status; backups verify artifact size and timestamp. Upgrading success from exit code to business assertion is the first cornerstone of cron observability.

Layer two: heartbeats are cheaper than alerts

Instead of waiting for a job to fail before alerting (a job that never ran, a dead machine, or a deceased cron daemon never reaches your alert logic), flip it: have the job report a heartbeat on success. Think dead man's switch - if the monitor doesn't receive a ping within N hours, the job isn't alive, and it notifies you proactively. GitHub Actions has hosted options; self-hosted, one small HTTP endpoint plus one scanning cron is enough.

Heartbeat essentials: append one curl or script call at the end of each job; set the timeout threshold to schedule interval times two plus buffer; include job name, date, and key output counts in the report so triage doesn't require SSH.

Layer three: keep a human-readable run archive

Beyond machine alerts, append one summary line to a fixed location each run: time, job, status, duration, output count. A file beats a database for 2 AM triage - tail reveals the past week's rhythm at a glance. The day duration doubles or output halves, the trend is visible to the naked eye, often earlier than any threshold alert.

Pre-launch self-check

Run every new cron job through four questions: Who knows when it fails? Can it silently succeed? Will retries cause duplicated side effects (dedupe before writes, make publishing idempotent)? Can logs pin the failing step within 30 seconds? Answer all four, and the job has earned the word unattended.

The dignity of automation isn't how smoothly it runs on ordinary days - it's that the moment it dies, someone, or at least something, is worrying on your behalf.