Files
multica/server
Bohan Jiang bfe9bf3eea feat(daemon): force-stop hung agent runs via idle watchdog (MUL-2281) (#2691)
* feat(daemon): force-stop hung agent runs via idle watchdog (MUL-2281)

A backend whose subprocess hangs on a stuck child process (e.g. claude
blocked on `docker ps` against a frozen dockerd) keeps the daemon's run
record at status="running" until the full DefaultAgentTimeout (2 h)
expires, because cmd.Wait() never returns and Session.Result is never
written. MUL-2225 spent 17+ minutes in this state in the wild.

Add a per-task idle watchdog around executeAndDrain:

- Wrap the caller's ctx so a single cancel propagates to the agent
  subprocess (via the ctx passed to backend.Execute) AND the drain loop.
- Stamp lastActivityAt every time the drain loop receives a message.
- Tick at window/2; when idle_for >= window AND session.Messages buffer
  is empty, set a fired flag and call cancel.
- Tag the resulting Result.Status as "idle_watchdog" so runTask routes
  it through a dedicated failure_reason instead of "agent_error".

Default window is 5 min, configurable via MULTICA_AGENT_IDLE_WATCHDOG;
set to 0 to disable. Tests cover the activity-then-silence case, the
zero-message case, the disabled case, and the happy path.

Co-authored-by: multica-agent <github@multica.ai>

* fix(daemon): skip idle watchdog while a tool call is in flight

A legitimate long-running tool call (npm install, docker build, test
suite) can sit silent between tool_use and tool_result for many minutes.
Without this gate, the watchdog would yank the agent mid-build.

Track unmatched tool_use messages in an atomic counter; only let the
watchdog fire when the counter is zero. tool_result clamps non-negative
so a stray result with no matching use can't re-arm the watchdog one
call too early.

Adds two regression tests:
  - DoesNotFireDuringInFlightToolCall: tool_use -> silence past
    window -> tool_result -> completed (must NOT fire)
  - FiresAfterToolResultIfBackendStaysSilent: tool_use -> tool_result
    -> silence past window (MUST fire — backend really is stuck)

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: multica-agent <github@multica.ai>
2026-05-15 19:48:39 +08:00
..