ACP Chain Cascade Failure: How We Traced the Bug Three Layers Deep

The Chain Structure

Last week our ACP chain was down for a full day. On the surface: an agent wasn't responding. Under the surface: three layers of overlapping bugs, each hiding the one below it.

Our ACP chain is structured as: OpenClaw → acpx-wrapper → acpx 0.3.1 → claude-agent-acp (0.24.2). A breakdown anywhere in this chain produces the same symptom: the agent isn't responding. Diagnosing which layer is broken requires working through each one systematically.

Layer 1: The Visible Symptom

Agent not responding to tasks. Check PM2: process running. Check logs: no recent activity, no errors. The absence of errors was the first misleading signal — it suggested the process was healthy when it was actually stuck.

Investigation: send a test task and watch the request trace. The task entered the queue and... nothing. No error, no processing, no timeout. Silent stall.

Layer 2: The First Real Bug

Digging into acpx logs: a connection to the upstream service had been established and never closed — a connection leak. The wrapper was waiting for a response that was never going to arrive, blocking all subsequent requests. acpx 0.3.1 has a known issue with connection pool exhaustion under specific timeout conditions. Fix: restart acpx, which cleared the leaked connection.

At this point we thought we were done. We weren't.

Layer 3: The Root Cause

After restarting, the agent came back but with degraded performance — tasks completing but taking 3-4x longer than baseline. Deeper trace: the API key being used had been rate-limited. Not blocked — rate-limited. Requests were going through at reduced priority, creating the performance degradation. The rate limiting was triggered by the connection leak recovery flooding the API with retry requests.

Fix: rotate to a secondary API key, wait for the rate limit window to expire on the primary.

The Lesson

Multi-layer failures have a pattern: the visible symptom almost never points to the root cause. Work through each layer systematically rather than fixing the first thing that looks wrong and assuming it's resolved. "It's back to normal" is not "the root cause is fixed."