Multica Eve a69d969fb1 fix(sweeper): gate running-task wall clock on runtime liveness (MUL-4107) (#4978) ...

The server-side running-task sweeper failed rows purely on `started_at >
now() - runningTimeoutSeconds` (2h30m). By its own comment the wall clock
is "mainly for runs whose daemon died without reporting" and "only needs
to sit generously above any realistic single run" — but the predicate
does not actually distinguish a healthy long-running task from an
orphaned one. On self-hosted deployments this kills multi-hour research
/ training runs mid-flight even though the daemon is still heartbeating
and the run is actively producing output.

The daemon side is intentionally unbounded (only inactivity watchdogs:
idle 30m, tool 2h); the server backstop was silently the only wall
clock. `FailStaleTasks` is now AND-gated on runtime liveness:

  * dispatched — unchanged; already excludes rows with a live
    `prepare_lease_expires_at` (renewed every 15s by the daemon between
    claim and StartTask).
  * running — new: excluded when the task's `agent_runtime` row is
    `online` AND `last_seen_at` is within the runtime stale window
    (staleThresholdSeconds = 150s, the same signal
    sweepStaleRuntimes already uses).

Healthy long-running tasks on live daemons are no longer killed by the
wall clock. The daemon-dead case remains primarily handled by
sweepStaleRuntimes in the same tick (Redis LivenessStore + DB stale +
FailTasksForOfflineRuntimes); the wall-clock branch is now a defensive
backstop for the pathological case where a runtime row lingers online
with a stale DB heartbeat for longer than the wall clock. `runtime_id
IS NULL` is treated as "not proving liveness" so the wall clock still
fires on that (rare / historical) shape.

The 2h30m default is unchanged — this is a gate, not a threshold
change. Tests updated: 4 existing running-task tests now age out the
runtime so they still exercise the wall clock; 2 new tests cover both
new invariants (healthy runtime → skipped; stale runtime → still
killed).

Fixes #4958

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>

2026-07-06 16:29:02 +08:00

125 KiB

Raw Permalink Blame History

View Raw