How to Design a Resumable Checkpoint for a Long-Running Task
"Resuming" does not mean "running it all over again." It means continuing from some intermediate state and making the result identical to an uninterrupted ru…
- Added to the lab
- Downloads
- 0
- Human review
- Review information not provided
"Resuming" does not mean "running it all over again." It means continuing from some intermediate state and making the result identical to an uninterrupted run. Before designing a checkpoint, answer three questions: where are we now, what has already been produced, and is it safe to redo. These three questions determine every trade-off that follows.
First, distinguish progress from state. Progress is "we have read up to record N"; state is "how far the output has been written, and which side effects have already occurred." Storing only an offset is the easiest thing to get wrong: record N was processed but the output never reached disk, so on resume it gets skipped and the result ends up one record short. The safer approach is to record a "committed" watermark and advance it only after the output is durable. For example, write a batch of records into the target table and commit, then update the checkpoint row in the same transaction. If the target supports upsert by primary key, you can also make the write itself idempotent and leave the checkpoint as a rough marker only.
Idempotency is the core. Every unit of work should be repeatable without producing duplicate side effects: upsert by business key, write files using "temp file + atomic rename," pass an idempotency key when calling an external API, or persist the response locally before continuing. If the external system does not support idempotency, you must weigh whether the checkpoint goes before or after the call: placing it before may cause duplicate calls; placing it after may lose calls. Usually you choose "at least once + deduplication on the consumer side."
The checkpoint itself must also be written atomically. Common approaches are to write a temporary file, fsync it, and rename it over the original; to do a single-row upsert in a database; or to use an append-only log with periodic snapshots. Never edit the main file in place while writing: after a crash, a half-written JSON file cannot be parsed. The file should carry a schema version, task ID, input data fingerprint, timestamp, and checksum, so that resume has something to judge by.
Granularity is a balance between cost and loss. Writing a checkpoint for every record can double the task's runtime through I/O amplification; writing one every hundred thousand records means a crash throws away a hundred thousand records' worth of work. A more practical approach is to commit by batch, with a batch size whose per-batch time falls between tens of seconds and a few minutes, aligned with the downstream write batch size. For example, 5,000 records per batch: process within the batch, and at the end of the batch commit the output and the checkpoint in one go.
On resume, do not blindly trust the checkpoint. First verify whether the input has changed (the file was replaced, rows were added to or removed from the upstream table), then verify whether the output is complete (record count, checksum), and also check whether the checkpoint version is compatible. If it is not compatible, either rerun from scratch or write migration logic. A crash can also leave "orphans": half-written output files and uncommitted rows. Clean them up before resuming, or have the downstream overwrite idempotently.
Multi-stage tasks should treat stage boundaries as checkpoints. Write a checkpoint as extraction, transformation, loading, and validation each complete, and subdivide within each stage; at the entrance to each stage, first confirm the upstream artifacts are ready. Do not let one giant function handle all the logic, or the checkpoint can only sit at the outermost layer and resume granularity becomes very coarse.
Testing recovery is worth more time than designing it. Kill -9 at random points, then compare the final result after resume with an uninterrupted run; you can also inject errors to simulate failure. Focus on the window "output written, checkpoint not yet written" — that is where mistakes most easily happen.
Also acknowledge the limits. Checkpoints cannot converge uncertainty: external API responses may change, time windows such as "the last 24 hours" have already slid by the time you resume, and random numbers, UUIDs, and current timestamps will make a rerun produce different results. When concurrent tasks share writable state, resume semantics become complicated. A checkpoint is also not a backup; it records only progress, it does not protect data, and disk corruption still requires an independent backup. Moreover, frequent fsync and checksums have real overhead on very large tasks, which needs to be measured rather than guessed.
Finally, write the definition of "resumable" and the invariants clearly, then choose the storage form, then set the granularity, and finally verify with kill tests. The essence of checkpoint design is to drive uncertainty into a few explicit, checkable boundaries.