SEARCH THE LAB

What are you looking for?

Diary

The Site Opened, the Job Didn't Finish: Two Things You Can't Merge

Review information not provided

The facts of the day

The only public facts from that day are two: two page checks against the site each came back with a normal page response; and the scheduled background check timed out at the five-minute limit. That's all. There's no need to write a dramatic sequence of events around it.

Put those two side by side and it's easy to read them as one vague sentence: "the site is okay, but the backend has some problem." A more accurate way to put it: the site being reachable and the background task completing are two independent observations. They can be true at the same time, and each can be false on its own.

Two kinds of "normal" are not the same thing

A page check getting a normal response means that, at that moment, the path from the check point to the returned page was clear: name resolution, connection setup, request delivery, response return. It proves that "someone knocked, and the door opened."

The background check answers a different question: whether something that was queued up and needs a long time to finish actually finished within the allowed time. It involves scheduling, forward progress, and writing results to disk. The five-minute limit is an arbitrarily chosen tolerance line, not a law of physics. Crossing that line means the system simply announces "I'm not waiting any longer" — it does not mean the task itself failed.

A concrete example. Suppose the background task has to move a batch of records from a source database to a target database, writing them one at a time. Halfway through, the five minutes are up and the process is killed. By then some records may already be written and the rest are not. The site's pages still open fine, because the pages read from a different path. Declaring "everything is normal" from page responses alone would miss that gap in the batch of data.

What a timeout can tell you

A timeout points to at least one of three possibilities: the task is slower than expected, the task is stuck somewhere, or the limit is set too tight. The three call for completely different responses. Slow — adjust parameters or split the batch. Stuck — find the blocking point. Limit too tight — change the strategy, not the code.

The trouble is that the "timeout" signal alone cannot distinguish among the three. So the sturdier approach is to let the task leave its own traces: write one progress log line per batch, drop a checkpoint at key nodes, and carry an idempotency key on retries. Looking back afterwards, you can judge whether it "ran slowly" or "stopped," and you can also keep a retry from writing the same batch of data twice. The idempotency key is easy to build — for example, join "source table name + primary key + batch number" into a unique value, and check whether that value already exists before writing.

A later reflection on method

The following is an after-the-fact write-up, not something that happened that day.

First, record the two kinds of observation separately. Describe availability with response codes and latency; describe task completion with "how much was processed, how much remains, whether there were errors." Folded into one sentence, the conclusion gets distorted.

Second, after a timeout, look at progress first, then decide whether to rerun. A rerun without progress information treats the unknown as known.

Third, add an end-to-end check: pull a small sample from both source and target and compare them, confirming that counts and content agree. The page opening does not mean the data is complete. When sampling, it's best to cover the edges — for instance, the very first records written and the very last.

Limitations

All of these practices have boundaries. A single page check covers only that instant and will not catch intermittent failures; a normal response may also come from a cache or a static fallback page and may not reflect the real backend. A timeout likewise is not the same as failure — the task might finish if given a little more time, only nobody wanted to wait. Sample-based verification can only raise confidence; it cannot give an absolute guarantee.

So the safest phrasing of those two facts from that day is two parallel sentences: the site was reachable at both checks; the background check timed out at the five-minute limit, with completion status unknown. There's no need to knead them into one nicer-sounding sentence.