Task Stuck at 61% for Four Hours Undetected: Giving Long-Running Tasks a Heartbeat

Last week, one of our batch pipelines stalled. It started running at 3 AM. When I checked the dashboard at 11 AM, the progress was stuck at 61%. The task status still showed `running`, the last log line was timestamped at 03:42, the process was alive, but CPU usage was 0%.

Digging deeper, we found that the SDK’s internal retry layer had hung the request. The outer-level timeout of 600 seconds was fine, but internal retries stacked up layer by layer, accumulating three hours of runtime, with each attempt technically “compliant.” The process was alive, the status was alive, but the task was effectively dead.

Since then, we’ve implemented three measures. None were expensive, but if they had all been in place that day, someone would have been paged within twenty minutes.

1. Heartbeats: Don’t Ask “Are You Alive?” Ask “What Are You Doing?”

The first thing we discarded was status-based liveness checks. `running` is a state, not proof. Heartbeats are proof: at the end of each stage, the task writes a line to a heartbeat file like `stage=xxx p=61 eta=42m at=`, including the progress percentage and a real timestamp.

The watchdog checks two conditions every 10 minutes:

- If the heartbeat hasn’t refreshed in over 15 minutes → mark as stalled

- If the progress number hasn’t changed for 30 consecutive minutes, but the file is still being written to → mark as idle/spinning

Separating these two cases is intentional. The former is “playing dead” and triggers an immediate page; the latter is “one link is slow but still moving,” which is lower severity and only logged in the daily report. After this separation, the number of pages dropped by three-quarters.

2. Layered Timeouts: Stalls Are Often Not the Outer Layer’s Fault

During that night’s stall, the outer 600-second timeout never triggered because internal retries kept stacking up. The lesson: timeouts must clearly specify which layer they apply to.

- Single HTTP request: 90-second hard timeout. On timeout, throw an error immediately without inner-layer retries.

- Single business stage: 15 minutes. First trigger sends an alert; second trigger kills the task.

- Entire task: 6 hours. Log only, do not kill.

We don’t kill at the third layer because interrupting a task mid-way can often be recovered via checkpoint resumption (as discussed in the article from 2026-08-21). Killing it outright would escalate a “stall” into a “total loss.”

The real time-saver was putting all three timeout values into a single configuration file and asserting in tests that “inner layers must be strictly shorter than outer layers.” Relying on configuration constraints is more reliable than relying on human memory—someone will inevitably loosen a setting, breaking the entire protection scheme.

3. Two-Step Process Termination: Send SIGTERM and Wait 30 Seconds

Previously, we killed tasks with `kill -9`. The problem was that some tasks would be halfway through writing a checkpoint, leaving corrupted files. The next resume attempt would crash immediately due to the bad checkpoint.

Now, terminating a process is a two-step critical change:

1. Send `SIGTERM`. The task’s signal handler stops and performs one final action: mark the current stage as `interrupted` and rename any incomplete checkpoint files with a `.tmp` extension.

2. If the process is still alive after 30 seconds, send `SIGKILL`.

During those 30 seconds, the process can still perform cleanup. This is the key difference from a bare `kill -9`. It seems trivial, but it basically turned “resume failure after stall” into “resume succeeds on the first try.”

What to Do After Detecting a Stall

When a stall is detected, do not let the watchdog automatically initiate recovery. The reason is that the watchdog runs on the same machine as the business logic. When a stall occurs, the machine is often already in a strange state—full memory, full disk. Automatic recovery risks falling into the same trap.

Our workflow: Detect stall → Push notification with a single line of context (stage, last progress, how long the heartbeat has been stale) → On-call engineer chooses one of two actions: resume or discard. The decision takes at most three seconds because all necessary information is included in the notification.

In Summary

- `running` is a state; the heartbeat file is the proof.

- Write layered timeouts, ensuring inner layers are strictly shorter than outer ones, backed by configuration assertions.

- Give processes a 30-second window for cleanup before killing; avoid bare `kill -9`.

- The watchdog only pages humans; it does not self-heal. Decisions must be executable within three seconds.

Since implementing this system, the same pipeline has never again suffered an “eight-hour undetected stall” incident. The longest stall we’ve seen since then took just 18 minutes from occurrence to the on-call engineer seeing the notification on their phone.