mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-27 13:06:20 +02:00
0b1333fb00
* feat(server): orphan-task recovery + auto-retry + manual rerun (MUL-1128)
When the daemon process crashed mid-task the issue was stuck at
in_progress for up to 2.5h: the in-flight task timeout was the only
mechanism that ever moved the row, and the runtime heartbeat sweeper
only fires after the runtime stays offline for 45s — a quick restart
beats both windows.
This change implements the A+B plan from the issue thread:
A. lifecycle hygiene
- migration 055 adds attempt / max_attempts / parent_task_id /
failure_reason / last_heartbeat_at to agent_task_queue
- new daemon-auth endpoint POST /runtimes/{id}/recover-orphans:
daemon calls it on every register so the server fails any
dispatched/running tasks the previous process left behind
- new daemon-auth endpoint POST /tasks/{id}/session: persists the
agent's session_id + work_dir mid-flight so a crash doesn't
lose the resume pointer (claude+codex emit MessageStatus with
SessionID; daemon forwards on the first one it sees)
- FailAgentTask / FailStaleTasks / FailTasksForOfflineRuntimes
now set failure_reason ('agent_error' / 'timeout' /
'runtime_offline')
B. auto-retry with resume context
- TaskService.MaybeRetryFailedTask spawns a fresh queued attempt
carrying parent's session_id/work_dir when the failure reason
is infrastructure-shaped (timeout, runtime_offline,
runtime_recovery) and attempt < max_attempts; skips autopilot
- wired into the runtime sweeper paths and TaskService.FailTask
so the user transparently sees a new in_progress run instead of
a stuck row
- new user-auth POST /api/issues/{id}/rerun + multica issue rerun
CLI for the manual escape hatch
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(server): address PR review for orphan-task recovery (MUL-1128)
Three review-must-fix items on top of the A+B implementation:
1. recover-orphans now funnels through TaskService.HandleFailedTasks,
the same shared post-failure pipeline used by the runtime sweeper.
This guarantees task:failed events are emitted, agent status is
reconciled, and issues stuck in_progress with no remaining active
task are reset to todo even when no auto-retry is created
(max_attempts exhausted, autopilot, non-retryable reason).
2. RerunIssue now uses CancelAgentTasksByIssueAndAgent, scoped to the
issue's current assignee. The previous implementation called
CancelAgentTasksByIssue, which would collateral-cancel parallel
@-mention agents on the same issue.
3. GetLastTaskSession now considers both completed and failed tasks
(mirroring GetLastChatTaskSession), ordering by the most recent
timestamp. With UpdateAgentTaskSession pinning session_id/work_dir
mid-flight, an auto-retry or manual rerun of a daemon-crash failure
now actually resumes the prior conversation context instead of
starting fresh — matching the stated B-branch behaviour.
go build / go vet pass; the existing service and agent test suites pass.
runtime_sweeper / handler integration tests require a local DB with the
055 migration (and the pre-existing 050 first_executed_at column).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
27 lines
1.4 KiB
SQL
27 lines
1.4 KiB
SQL
-- Adds task-level retry/lease bookkeeping so the runtime sweeper and the
|
|
-- daemon startup recovery path can distinguish "fresh attempt" from
|
|
-- "auto-rerun after orphan", and so resume context survives a daemon
|
|
-- restart mid-execution.
|
|
--
|
|
-- Columns:
|
|
-- attempt -- 1 for the first run, incremented per auto-retry/manual rerun
|
|
-- max_attempts -- ceiling honored by the auto-retry path; 1 disables retry
|
|
-- parent_task_id -- back-pointer to the task that this one re-attempts
|
|
-- failure_reason -- coarse classifier set when status flips to failed:
|
|
-- 'agent_error', 'timeout', 'runtime_offline',
|
|
-- 'runtime_recovery', 'manual'. The auto-retry path
|
|
-- uses this to decide whether to spawn a child task.
|
|
-- last_heartbeat_at -- mid-task heartbeat timestamp; the runtime heartbeat
|
|
-- already drives runtime liveness, but per-task
|
|
-- timestamps let us tell stale tasks apart from
|
|
-- long-running ones in future enhancements.
|
|
|
|
ALTER TABLE agent_task_queue
|
|
ADD COLUMN attempt INT NOT NULL DEFAULT 1,
|
|
ADD COLUMN max_attempts INT NOT NULL DEFAULT 2,
|
|
ADD COLUMN parent_task_id UUID REFERENCES agent_task_queue(id) ON DELETE SET NULL,
|
|
ADD COLUMN failure_reason TEXT,
|
|
ADD COLUMN last_heartbeat_at TIMESTAMPTZ;
|
|
|
|
CREATE INDEX idx_agent_task_queue_parent ON agent_task_queue(parent_task_id);
|