* feat(task): wire claim lease queries into TaskService and sweeper (MUL-2246)
- ClaimTask now uses ClaimAgentTaskWithLease (generates claim_token + lease)
- StartTask accepts optional claim_token for token-verified start
- AgentTaskResponse includes claim_token for daemon to use
- Daemon client sends claim_token in StartTask body
- Sweeper calls RequeueExpiredClaimLeases each tick
- Legacy daemons without claim_token still work (graceful fallback)
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): address PR #2662 review blockers (MUL-2246)
1. ClaimAgentTaskForRuntime: push runtime_id into atomic SQL WHERE clause
so runtime A cannot claim tasks queued for runtime B under the same agent.
2. Legacy StartAgentTask: add claim_token IS NULL guard so leased rows
cannot be started without token verification. Handler rejects malformed
tokens with 400 instead of silently degrading to legacy path.
3. StartAgentTaskWithClaimToken: validate claim_expires_at >= now(),
preserve claim_token until terminal state (only clear claim_expires_at),
use CTE + UNION ALL for idempotent retry when daemon resends after a
lost StartTask response. Return 409 Conflict on token mismatch/expiry.
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): StartTask 409 handling, transport retry, claim_token on FailTask (MUL-2246)
- StartTask 409 (claim superseded): release slot, don't call FailTask
- StartTask transport timeout/5xx: retry once with same token, then
check task status before failing
- FailTask now sends claim_token; server-side FailAgentTask SQL adds
AND (claim_token IS NULL OR claim_token = @claim_token) guard so
stale daemons cannot fail tasks that have been re-claimed
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): close FailTask token bypass and RequeueExpiredClaimLeases liveness gap (MUL-2246)
Blocker 1 - FailTask token validation:
- SQL: change (param IS NULL OR claim_token = param) to
(param IS NULL AND claim_token IS NULL) OR claim_token = param
so tokenless requests can only fail legacy (tokenless) rows.
- task.go: malformed claim_token now returns ErrInvalidClaimToken (400)
instead of being silently dropped to NULL.
- Handler: maps ErrInvalidClaimToken→400, ErrClaimTokenInvalid→409.
- Service: when UPDATE returns no rows but task is still active,
return ErrClaimTokenInvalid (token mismatch) instead of silent success.
Blocker 2 - RequeueExpiredClaimLeases runtime liveness:
- SQL: JOIN agent_runtime, only requeue tasks where runtime is 'online'.
Dead/offline runtime tasks stay dispatched for FailTasksForOfflineRuntimes.
- FOR UPDATE → FOR UPDATE OF atq (required with JOIN).
Regression tests:
- task_claim_token_test.go: malformed, tokenless-on-tokened, wrong-token
- requeue_lease_test.go: SQL must JOIN agent_runtime with online filter
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): move expired lease requeue to ClaimTaskForRuntime preflight, add heartbeat freshness backstop (MUL-2246)
- Add RequeueExpiredClaimLeasesForRuntime: per-runtime preflight self-requeue
in ClaimTaskForRuntime. Runtime proves liveness by actively claiming, so no
heartbeat check needed.
- Update global RequeueExpiredClaimLeases to require ar.last_seen_at freshness
(stale_threshold_secs param). Prevents requeuing to a dead runtime in the
90s gap between lease expiry (60s) and offline detection (150s).
- Add regression tests verifying the heartbeat freshness check and that the
preflight query does not join agent_runtime.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): use LivenessStore for global requeue, move preflight before empty-cache (MUL-2246)
Blocker 1: Global RequeueExpiredClaimLeases now uses LivenessStore.IsAliveBatch
to verify runtimes are truly alive before requeuing expired leases. When
LivenessStore is unavailable (no Redis), global requeue is skipped entirely —
the preflight self-requeue in ClaimTaskForRuntime handles live runtimes. This
closes the 60-150s gap where a dead runtime still appears online in DB.
Blocker 2: Moved RequeueExpiredClaimLeasesForRuntime BEFORE EmptyClaim.IsEmpty
fast-path in ClaimTaskForRuntime. Expired leases are now requeued (which bumps
the empty cache via notifyTaskAvailable) before the empty check can
short-circuit the claim path.
Also adds ListRuntimesWithExpiredClaimLeases SQL query and LivenessChecker
interface on TaskService.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): wire EmptyClaimCache into backend taskSvc for backstop requeue (MUL-2246)
The backend taskSvc used by the sweeper only had Liveness wired but not
EmptyClaim. When global backstop requeue called notifyTaskAvailable,
s.EmptyClaim.Bump() was a nil no-op — the handler's empty-cache was never
invalidated, so the daemon's next claim hit a stale empty verdict.
Fix: wire the same Redis-backed EmptyClaimCache into the backend taskSvc
in main.go (same Redis keys as router.go:139 handler instance).
Add regression test verifying backstop requeue invalidates the handler's
empty-cache.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): global backstop must not requeue — alive runtimes use preflight, dead stay dispatched (MUL-2246)
- RequeueExpiredClaimLeases is now a no-op (returns 0 always)
- Alive runtimes self-requeue via ClaimTaskForRuntime preflight
- Dead runtimes stay dispatched for FailTasksForOfflineRuntimes
- Rewriting to queued on dead runtime creates 2h blackhole (offline
sweeper only handles dispatched/running)
- Test actually calls RequeueExpiredClaimLeases and asserts 0 in all cases
Co-authored-by: multica-agent <github@multica.ai>
* fix(daemon): remove duplicate usage reporting block after merge conflict (MUL-2246)
The merge resolution introduced a second ReportTaskUsage call after the
status check, duplicating the usage-before-early-return block that already
runs right after runner.run. Remove the duplicate and add a regression test
asserting /usage is called exactly once on the normal completion path.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: multica-agent <github@multica.ai>
* feat(server): orphan-task recovery + auto-retry + manual rerun (MUL-1128)
When the daemon process crashed mid-task the issue was stuck at
in_progress for up to 2.5h: the in-flight task timeout was the only
mechanism that ever moved the row, and the runtime heartbeat sweeper
only fires after the runtime stays offline for 45s — a quick restart
beats both windows.
This change implements the A+B plan from the issue thread:
A. lifecycle hygiene
- migration 055 adds attempt / max_attempts / parent_task_id /
failure_reason / last_heartbeat_at to agent_task_queue
- new daemon-auth endpoint POST /runtimes/{id}/recover-orphans:
daemon calls it on every register so the server fails any
dispatched/running tasks the previous process left behind
- new daemon-auth endpoint POST /tasks/{id}/session: persists the
agent's session_id + work_dir mid-flight so a crash doesn't
lose the resume pointer (claude+codex emit MessageStatus with
SessionID; daemon forwards on the first one it sees)
- FailAgentTask / FailStaleTasks / FailTasksForOfflineRuntimes
now set failure_reason ('agent_error' / 'timeout' /
'runtime_offline')
B. auto-retry with resume context
- TaskService.MaybeRetryFailedTask spawns a fresh queued attempt
carrying parent's session_id/work_dir when the failure reason
is infrastructure-shaped (timeout, runtime_offline,
runtime_recovery) and attempt < max_attempts; skips autopilot
- wired into the runtime sweeper paths and TaskService.FailTask
so the user transparently sees a new in_progress run instead of
a stuck row
- new user-auth POST /api/issues/{id}/rerun + multica issue rerun
CLI for the manual escape hatch
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(server): address PR review for orphan-task recovery (MUL-1128)
Three review-must-fix items on top of the A+B implementation:
1. recover-orphans now funnels through TaskService.HandleFailedTasks,
the same shared post-failure pipeline used by the runtime sweeper.
This guarantees task:failed events are emitted, agent status is
reconciled, and issues stuck in_progress with no remaining active
task are reset to todo even when no auto-retry is created
(max_attempts exhausted, autopilot, non-retryable reason).
2. RerunIssue now uses CancelAgentTasksByIssueAndAgent, scoped to the
issue's current assignee. The previous implementation called
CancelAgentTasksByIssue, which would collateral-cancel parallel
@-mention agents on the same issue.
3. GetLastTaskSession now considers both completed and failed tasks
(mirroring GetLastChatTaskSession), ordering by the most recent
timestamp. With UpdateAgentTaskSession pinning session_id/work_dir
mid-flight, an auto-retry or manual rerun of a daemon-crash failure
now actually resumes the prior conversation context instead of
starting fresh — matching the stated B-branch behaviour.
go build / go vet pass; the existing service and agent test suites pass.
runtime_sweeper / handler integration tests require a local DB with the
055 migration (and the pre-existing 050 first_executed_at column).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
