No Overnight Demo Runs: Moving Preflight Checks to CI After a Simulation Environment Freeze

Last month, we delivered an automation script for a simulation environment to a team. During the final batch run demonstration before delivery, the system crashed at 2:30 AM. The logs showed only a single timeout error, and the environment became completely unresponsive. The next morning, we spent hours reinstalling the entire dependency stack while staring at the screen. The post-mortem meeting was silent because this issue had supposedly been "prevented long ago"—it was explicitly listed in the runbook.

The problem lay with the runbook itself. That 40-step pre-run checklist sat in a shared document, labeled "Verify Before Execution." However, there was a slight misalignment in understanding what "verified" meant between the two people involved: the person who wrote the script performed the pre-check, but the colleague giving the demo actually clicked "run." The document stated that the simulator's time zone must align with the log time zone. The checker verified this against their local machine time, failing to account for the fact that the demo machine was hosted in an overseas data center with an 8-hour time difference. When the timeout occurred in the early hours, the checker was already home asleep.

Our change was minimal: **we moved the checklist from the shared document into the code repository, implementing it as a mandatory green-light preflight gate in our CI pipeline.**

1. Each simulation configuration file includes a `preflight.json`, declaring the time zone, data paths, and dependency versions.

2. The CI runs a dry-run: it loads the configuration, constructs the pipeline, and executes two steps without writing to disk, simply verifying "whether this path is viable."

3. The dry-run output (single-step duration, sample data count) is automatically posted to the Pull Request description. Merging is blocked until this passes, ensuring only validated configs enter the formal scheduling queue.

4. During scheduling, the scheduler compares the actual dry-run duration against the budget. If it exceeds the limit, the job is rejected immediately.

A second benefit is smoother handovers. When the colleague responsible for the demo took over, they no longer had to guess why we had configured things a certain way. Each configuration is accompanied by its actual dry-run metrics, which are far more reliable than verbal handovers.

The third benefit is that **it forces the runbook to slim down**. Of the original 40 steps, 30 could be automatically checked. After moving these into CI, only 8 steps remained for manual verification. Three of those were related to client business logic rather than engineering, so we later moved them to the client’s own wiki.

**Pitfall encountered:** Initially, the dry-run used mock data. However, the real batch data contained an unusually long sample, causing the dry-run to pass while the actual run timed out. We switched to using "real data headers + random padding," and the issue never recurred. The cost was an extra 20 minutes spent building the data pipeline, but that was far cheaper than reinstalling the environment at 2 AM.

By the Numbers

- Delivery cycle reduced from 11 days to 7 days (saving two instances of freezing and reinstalling).

- Pre-demo checks dropped from 40 manual steps to a single CI status check.

- The `preflight.json` now uses a shared schema across 12 configurations; the check cost for adding a new task is approximately 15 minutes.

The lesson is simple: **Checklists written for others to read lose effectiveness when merely skimmed; checks built into the pipeline and executed automatically do not.** Not every step is suitable for automation, but moving the suitable ones early allows the runbook to become short enough that people will actually read it.