After the Automation Script Retried Itself to Death for the Third Time, We Separated Observation from Execution

Last month, during the 2 AM release window, the same bug triggered three alerts. The first time, the script created its own mess and cleaned it up. The second time, it swallowed the exception and returned an empty result. The third time, it detected the files it had written itself, interpreting that detection as a condition for retrying, resulting in forty minutes of idle spinning. It wasn’t until we moved the "decision to retry" logic out of the execution process that this livelock finally stopped. Here’s a record of what happened.

Three Ways to Die

The first failure was straightforward: the execution script failed completely during the V4 database ingestion step due to a schema conflict. It caught the exception, cleaned up the temporary directory, and exited with code 0, leaving a single line of red text in the logs. Although it failed, the troubleshooting path was clear.

The second failure was more insidious: in the same code block, the exception was wrapped by the translation conversion layer and returned as an empty object. The executor treated this empty object as a normal result, producing neither a diff nor an error message. The task showed as "Completed," and the report normally recorded the three-language review. This was a false success, which is much harder to detect than a outright failure.

The third failure finally broke the cycle: in each polling round, the script rebuilt the temporary directory and wrote the previous round’s failure output into a re-check file. The re-check logic detected the existence of this file and triggered cleanup again, which in turn repopulated the re-check file. The judgment logic at the start of the execution process and the actions at the end locked onto each other, spinning idly for forty minutes, with every round reporting the same message: "Cleanup complete."

Where Troubleshooting Got Stuck

The biggest hurdle wasn’t locating the issue, but the silence: the task showed as completed, the report had records, and monitoring showed no anomalies. The real signal was hidden outside the report—the modification time of the re-check file was earlier than the trigger time. The same rule appeared twice in the file: once as a judgment condition and once as the product of an action.

More embarrassingly, I was the one who originally requested the "one-click cleanup" feature. The rationale was simple: use a single command to clean up residues from the previous round. The requirement itself was sound, but with the observer and the executor residing in the same process, what was observed was always an echo of the execution result—a sensor that only reflects its own state. This structural pitfall wasn’t our first stumble; the final reckoning cost us an entire prime-time release window.

Guardrail Design

Only after separating the components could we properly discuss fixes. The separation was a one-time structural change that allowed for gradual improvements:

**Separate observation from triggering.** Temporary files are updated with a completion marker by the writer during atomic writes. The retrier only reads this completion marker to make decisions. Since it is read-only, it will never re-trigger actions.

**Every temporary artifact carries an owner and a TTL (Time-To-Live).** Files from different sources no longer overwrite each other. If the owner doesn’t refresh within the TTL, the next stage skips it regardless of whether the owner is still alive, handing it over to an independent expiration garbage collector.

**Diverge empty results from errors.** The conversion layer is prohibited from wrapping exceptions into empty objects. Empty results must go through an independent diff branch. The report must either contain body content or include a diff explaining why the result is empty.

**Add a "Change Count" column to the report.** For rounds where execution occurred but nothing changed, this column shows zero. Zero isn’t necessarily wrong, but it should be reviewed by a human, not glossed over by the words "Task Completed."

For the time being, risk coverage is backed by three routing-layer coverage checks (these three judgments were also fully migrated to independent processes). After the fix, we ran continuously for three days. There was only one hint of similar idle spinning, which was blocked by the "Change Count is Zero" alert before any reader saw it. In the round that truly required changes, the diff pointed out specific schema fields, confirming it was not idle spinning.

In One Sentence

If your execution script can verify its own output, it can issue itself a pass. The verifier must be independent of the executor, and observation must be independent of execution—no one is allowed to live inside the signals they create themselves.