Three Lessons We Learned About Checkpointing from a Batch Job That Died at 98%

Last month, we ran a batch job on our local inference server: classifying 24,000 anonymized customer service messages, with each message processed by a local model. A single run took a full four hours.

Trouble struck in mid-August. During the third run, a voltage fluctuation in the server room caused the machine to reboot abruptly when it had processed 23,500 records. Only 500 records remained—a task that should have taken ten minutes. But what actually happened? We stared at an empty result file for a long time. Not a single byte of the 8GB of intermediate data had been written to disk because the script was designed to "write everything at once upon completion." Four hours of work, lost. We had to start over from scratch.

After that incident, we broke down the concept of checkpointing (resume-from-breakpoint) into three key practices, each learned through painful experience.

**First: Write results in small batches; never trust "save at the end."**

Our current approach appends to a JSONL file every 100 records and explicitly calls `flush`. We arrived at 100 through experimentation: the overhead of a single checkpoint is negligible, and in the worst-case scenario, the amount of rework is limited to about 20 minutes. We deliberately avoided aiming for minimal checkpoint intervals. While writing to disk every 10 records might feel safer, performing over 900 `fsync` operations increased the total wall-clock time of the job by approximately 7%. In the end, you pay for every millisecond. Initially, we used SQLite to track progress, but after one full run, the disk was filled up by logs, leaving the database with numerous half-committed transactions. It wasn't worth the trouble. Append-only streaming files turned out to be the simplest solution—if it breaks, it breaks, but as long as it works, it works.

A quick note on where to write these files: mount the results directory and the logs directory on separate disks. We once mounted both on the same data disk and ran into issues: a log flush consumed half the inodes on the partition, causing the batch job to mysteriously report EIO errors. It took us two hours of troubleshooting to realize that we had exhausted the inodes, not that there was anything wrong with the data itself.

**Second: Determine resume progress solely based on result files, not in-memory counters or log percentages.**

The lesson from our first failure was that progress numbers are unreliable. The logs might say "19,200/24,000," but the data file may have only been flushed up to 19,100. Our current recovery logic is simple: read the result file, collect the IDs of completed items into a set, and define pending tasks as Total − Completed. It doesn't matter how long the job ran or where it crashed; the set dictates the truth. The principle is straightforward: never store two copies of state that can be computed. In-memory counters, progress bars, and heartbeat percentages are all derived values. If they are lost, so be it.

There’s another common pitfall here: the last line of an append-only JSONL file might be incomplete. A power outage could interrupt a write halfway through a line. During recovery, you must verify the integrity of the last line; if it’s corrupted, discard it and reprocess that specific record. We wrote a lint script for this purpose: it parses each line independently and reports the line number if parsing fails. This five-minute task has saved us from two much larger debugging sessions.

**Third: Idempotency. For the same input, no matter how many times you rerun it, there should only be one result.**

The unique key for a record isn’t a database auto-increment ID, but a hash of "input content + model version + prompt version." The advantage of this approach is that when the model is upgraded or the prompt is revised, the old checkpoint directory is automatically invalidated, and the new directory starts from zero. Old and new results will never get mixed up. We previously suffered from a situation where a prompt was quietly updated, causing old results to mix with the new batch. The resulting report contained half old-version and half new-version data, which was even harder to explain than simply rerunning the job.

Before going live, we added a "moment of truth" check to our recovery logic: randomly sample 200 IDs and confirm that they exist in the result file and that their content can be parsed successfully. The number 200 was arbitrary, but without this sample-level spot check, our belief that "the set equals the true progress" would be based on faith rather than evidence.

Let’s look at the numbers: After implementing these fixes, the second run of the task crashed again at 74%. Recovery took only 11 minutes to complete the remaining portion, with 3 minutes spent rescanning the set of completed items. Over the past three months, this mechanism has supported over twenty runs. There were two interruptions, both recovered within ten minutes, and neither required any recomputation.

Batch jobs are essentially like long-distance running: you don’t need to remember every kilometer, just leave a marker at every intersection.