mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-13 05:16:29 +02:00
* feat(autopilot): skip dispatch when assignee runtime is offline (MUL-1899) Prevents scheduled autopilots from accumulating doomed tasks against offline / archived / unbound agents. Before this change, a paused laptop or crashed daemon would let a 5-minute-cron autopilot pile up thousands of queued agent_task_queue rows that no runtime would ever drain — this is the dominant source of the 89k stuck-task backlog flagged in MUL-1899. DispatchAutopilot now performs a pre-flight admission check on the assignee agent's runtime status. If the runtime is not 'online' (or the agent is archived / has no runtime bound / has no assignee), the run is recorded as 'skipped' with a failure_reason and no task is enqueued. Skipped runs still emit autopilot:run.done so the UI / activity feed reflect that the trigger fired and was evaluated. Skipped runs are deliberately NOT counted toward the failure-ratio auto-pause: a user who closes their laptop overnight should not have their autopilot paused. Sustained server-side failures keep their existing pause path via the failure monitor. Tests: added an integration test that creates an offline runtime and asserts DispatchAutopilot records a skipped run with no task enqueued. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * feat(scheduler): expire stale queued tasks via TTL sweeper (MUL-1899) Companion to the dispatch-time admission gate added in this PR. The admission gate prevents *new* tasks from being enqueued against an offline runtime, but it does not drain the historical backlog (~89k stuck queued rows observed at MUL-1899 baseline) and does not help when a runtime goes offline *after* a task has already been queued. This adds a passive TTL sweeper: - New SQL query `ExpireStaleQueuedTasks` transitions queued tasks older than the TTL to status='failed' with failure_reason='queued_expired' and a clear error message. - Sweep is capped per tick (`queuedExpireBatchSize`, default 500) via a CTE+LIMIT so that draining a large backlog cannot monopolise the DB on a single tick. At 30s ticks the worst case is 60k rows/hour. - Wired into the existing 30s `runRuntimeSweeper` loop alongside `sweepStaleTasks` and reuses `taskSvc.HandleFailedTasks` so the expired tasks broadcast `task:failed` events, reconcile agent status, and roll back any in-progress issues — same lifecycle as any other failed task. - Default TTL = 2h. Conservatively above any reasonable "queued behind a long-running task" window (default agent timeout is 2h, sweeper runs every 30s) so legitimate work isn't expired. - Integration tests cover the happy path (stale → expired, fresh → left alone, correct status/reason/error) and the per-tick batch cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * fix(autopilot): address review blockers from PR #2311 (MUL-1899) GPT-Boy review of the offline-runtime + queued-TTL PR flagged four blockers; this commit addresses them all. 1. Restore the 'skipped' autopilot_run status in the DB constraint. Migration 043 had removed 'skipped' along with the now-defunct concurrency_policy feature, so the new admission gate's INSERT of status='skipped' violated `autopilot_run_status_check` and broke `TestAutopilotDispatchSkipsWhenRuntimeOffline` in CI. New migration 079 re-adds 'skipped' to the CHECK list. The down migration migrates skipped → failed before re-tightening, mirror- ing what 043 did for the original removal. 2. Make `ExpireStaleQueuedTasks` race-safe. The CTE-then-UPDATE pattern could clobber a task that the daemon claimed between victim selection and the outer update. Two guards added: - `FOR UPDATE SKIP LOCKED` in the CTE so we never wait on a row that's currently being claimed (and never block the claim path either). - The outer UPDATE now re-checks `t.status = 'queued'` AND the TTL predicate so even if a row's lock is released after a successful claim, we cannot transition a now-dispatched/ running task to 'failed'. 3. Add a partial index for the queued-TTL sweeper. `idx_agent_task_queue_queued_created_at` on `created_at WHERE status = 'queued'` — keeps the 30s sweep query (status=queued AND created_at < ... ORDER BY created_at LIMIT 500) cheap even when historical terminal rows accumulate (~89k+ at MUL-1899 baseline). The partial predicate keeps the index tiny because only in-flight rows live in 'queued'. 4. Fix the failure-monitor denominator. `SelectAutopilotsExceedingFailureThreshold` had been counting 'skipped' toward total runs, which would have diluted the failure ratio: a 100%-failing autopilot could mask itself behind a wall of admission skips. With 'skipped' restored as a real status, the auto-pause monitor must explicitly exclude it from BOTH numerator and denominator — admission skips are neither a success nor a failure. Verified: `go test ./cmd/server/... ./internal/service/...` passes (including TestAutopilotDispatchSkipsWhenRuntimeOffline, TestExpireStaleQueuedTasks, TestExpireStaleQueuedTasksRespectsBatch Limit). `go build ./... && go vet ./...` clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * fix(migrations): split queued-task TTL index into concurrent migration Per PR #2311 review: agent_task_queue is a hot table, so building the new partial index with plain CREATE INDEX inside migration 079 would hold ACCESS EXCLUSIVE on the queue and block dispatch during deploy. The migration runner does not allow CONCURRENTLY to share a file with other statements (documented in 068), so split the index into its own single-statement file 080 — matching the existing pattern in 035 / 067 / 074 / 075 / 078. Migration 079 keeps the autopilot_run constraint change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Eve <eve@multica-ai.local> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai>
254 lines
8.2 KiB
SQL
254 lines
8.2 KiB
SQL
-- =====================
|
|
-- Autopilot CRUD
|
|
-- =====================
|
|
|
|
-- name: ListAutopilots :many
|
|
SELECT * FROM autopilot
|
|
WHERE workspace_id = $1
|
|
AND (sqlc.narg('status')::text IS NULL OR status = sqlc.narg('status'))
|
|
ORDER BY created_at DESC;
|
|
|
|
-- name: GetAutopilot :one
|
|
SELECT * FROM autopilot
|
|
WHERE id = $1;
|
|
|
|
-- name: GetAutopilotInWorkspace :one
|
|
SELECT * FROM autopilot
|
|
WHERE id = $1 AND workspace_id = $2;
|
|
|
|
-- name: CreateAutopilot :one
|
|
INSERT INTO autopilot (
|
|
workspace_id, title, description, assignee_id,
|
|
status, execution_mode, issue_title_template,
|
|
created_by_type, created_by_id
|
|
) VALUES (
|
|
$1, $2, sqlc.narg('description'), $3,
|
|
$4, $5, sqlc.narg('issue_title_template'),
|
|
$6, $7
|
|
) RETURNING *;
|
|
|
|
-- name: UpdateAutopilot :one
|
|
UPDATE autopilot SET
|
|
title = COALESCE(sqlc.narg('title'), title),
|
|
description = COALESCE(sqlc.narg('description'), description),
|
|
assignee_id = COALESCE(sqlc.narg('assignee_id')::uuid, assignee_id),
|
|
status = COALESCE(sqlc.narg('status'), status),
|
|
execution_mode = COALESCE(sqlc.narg('execution_mode'), execution_mode),
|
|
issue_title_template = sqlc.narg('issue_title_template'),
|
|
updated_at = now()
|
|
WHERE id = $1
|
|
RETURNING *;
|
|
|
|
-- name: DeleteAutopilot :exec
|
|
DELETE FROM autopilot WHERE id = $1;
|
|
|
|
-- name: UpdateAutopilotLastRunAt :exec
|
|
UPDATE autopilot SET last_run_at = now(), updated_at = now()
|
|
WHERE id = $1;
|
|
|
|
-- =====================
|
|
-- Autopilot Trigger CRUD
|
|
-- =====================
|
|
|
|
-- name: ListAutopilotTriggers :many
|
|
SELECT * FROM autopilot_trigger
|
|
WHERE autopilot_id = $1
|
|
ORDER BY created_at ASC;
|
|
|
|
-- name: GetAutopilotTrigger :one
|
|
SELECT * FROM autopilot_trigger
|
|
WHERE id = $1;
|
|
|
|
-- name: CreateAutopilotTrigger :one
|
|
INSERT INTO autopilot_trigger (
|
|
autopilot_id, kind, enabled, cron_expression, timezone,
|
|
next_run_at, webhook_token, label
|
|
) VALUES (
|
|
$1, $2, $3, sqlc.narg('cron_expression'), sqlc.narg('timezone'),
|
|
sqlc.narg('next_run_at'), sqlc.narg('webhook_token'), sqlc.narg('label')
|
|
) RETURNING *;
|
|
|
|
-- name: UpdateAutopilotTrigger :one
|
|
UPDATE autopilot_trigger SET
|
|
enabled = COALESCE(sqlc.narg('enabled')::boolean, enabled),
|
|
cron_expression = COALESCE(sqlc.narg('cron_expression'), cron_expression),
|
|
timezone = COALESCE(sqlc.narg('timezone'), timezone),
|
|
next_run_at = sqlc.narg('next_run_at'),
|
|
label = COALESCE(sqlc.narg('label'), label),
|
|
updated_at = now()
|
|
WHERE id = $1
|
|
RETURNING *;
|
|
|
|
-- name: DeleteAutopilotTrigger :exec
|
|
DELETE FROM autopilot_trigger WHERE id = $1;
|
|
|
|
-- name: AdvanceTriggerNextRun :exec
|
|
UPDATE autopilot_trigger
|
|
SET next_run_at = sqlc.narg('next_run_at'),
|
|
last_fired_at = now(),
|
|
updated_at = now()
|
|
WHERE id = $1;
|
|
|
|
-- =====================
|
|
-- Autopilot Run Management
|
|
-- =====================
|
|
|
|
-- name: CreateAutopilotRun :one
|
|
INSERT INTO autopilot_run (
|
|
autopilot_id, trigger_id, source, status, trigger_payload
|
|
) VALUES (
|
|
$1, sqlc.narg('trigger_id'), $2, $3, sqlc.narg('trigger_payload')
|
|
) RETURNING *;
|
|
|
|
-- name: GetAutopilotRun :one
|
|
SELECT * FROM autopilot_run
|
|
WHERE id = $1;
|
|
|
|
-- name: ListAutopilotRuns :many
|
|
SELECT * FROM autopilot_run
|
|
WHERE autopilot_id = $1
|
|
ORDER BY created_at DESC
|
|
LIMIT $2 OFFSET $3;
|
|
|
|
-- name: UpdateAutopilotRunIssueCreated :one
|
|
UPDATE autopilot_run
|
|
SET status = 'issue_created', issue_id = $2
|
|
WHERE id = $1
|
|
RETURNING *;
|
|
|
|
-- name: UpdateAutopilotRunRunning :one
|
|
UPDATE autopilot_run
|
|
SET status = 'running', task_id = $2
|
|
WHERE id = $1
|
|
RETURNING *;
|
|
|
|
-- name: UpdateAutopilotRunCompleted :one
|
|
UPDATE autopilot_run
|
|
SET status = 'completed', completed_at = now(), result = sqlc.narg('result')
|
|
WHERE id = $1
|
|
RETURNING *;
|
|
|
|
-- name: UpdateAutopilotRunFailed :one
|
|
UPDATE autopilot_run
|
|
SET status = 'failed', completed_at = now(), failure_reason = $2
|
|
WHERE id = $1
|
|
RETURNING *;
|
|
|
|
-- name: UpdateAutopilotRunSkipped :one
|
|
-- Marks an autopilot_run as skipped without enqueueing any task. Used by the
|
|
-- pre-flight admission check when the assignee agent's runtime is offline:
|
|
-- creating an issue / task in that state would just pile a doomed job onto
|
|
-- agent_task_queue (the canonical "持续给离线 local agent 入队" symptom from
|
|
-- MUL-1899). Recording the skip + reason gives the UI / failure monitor / ops
|
|
-- a paper trail without polluting the failure ratio.
|
|
UPDATE autopilot_run
|
|
SET status = 'skipped', completed_at = now(), failure_reason = $2
|
|
WHERE id = $1
|
|
RETURNING *;
|
|
|
|
-- =====================
|
|
-- Scheduler Queries
|
|
-- =====================
|
|
|
|
-- name: ClaimDueScheduleTriggers :many
|
|
-- Atomically claim all due schedule triggers to prevent concurrent execution.
|
|
-- Joins the autopilot table to ensure only active autopilots are fired.
|
|
UPDATE autopilot_trigger t
|
|
SET next_run_at = NULL
|
|
FROM autopilot a
|
|
WHERE t.autopilot_id = a.id
|
|
AND t.kind = 'schedule'
|
|
AND t.enabled = true
|
|
AND t.next_run_at IS NOT NULL
|
|
AND t.next_run_at <= now()
|
|
AND a.status = 'active'
|
|
RETURNING t.*, a.workspace_id AS autopilot_workspace_id;
|
|
|
|
-- =====================
|
|
-- Task Queue (run_only mode)
|
|
-- =====================
|
|
|
|
-- name: CreateAutopilotTask :one
|
|
INSERT INTO agent_task_queue (agent_id, runtime_id, issue_id, status, priority, autopilot_run_id, trigger_summary)
|
|
VALUES ($1, $2, NULL, 'queued', $3, $4, sqlc.narg(trigger_summary))
|
|
RETURNING *;
|
|
|
|
-- =====================
|
|
-- Run lookup by linked entities
|
|
-- =====================
|
|
|
|
-- name: GetAutopilotRunByIssue :one
|
|
SELECT * FROM autopilot_run
|
|
WHERE issue_id = $1 AND status IN ('issue_created', 'running')
|
|
LIMIT 1;
|
|
|
|
-- name: FailAutopilotRunsByIssue :exec
|
|
-- Fails active autopilot runs linked to a given issue.
|
|
-- Must be called BEFORE issue deletion (ON DELETE SET NULL clears issue_id).
|
|
UPDATE autopilot_run
|
|
SET status = 'failed', completed_at = now(), failure_reason = 'linked issue was deleted'
|
|
WHERE issue_id = $1
|
|
AND status IN ('issue_created', 'running');
|
|
|
|
-- =====================
|
|
-- Scheduler Recovery
|
|
-- =====================
|
|
|
|
-- name: RecoverLostTriggers :many
|
|
-- Finds schedule triggers that were claimed (next_run_at = NULL) but never
|
|
-- advanced — typically due to a scheduler crash. Returns them so the scheduler
|
|
-- can recompute next_run_at.
|
|
SELECT t.*, a.workspace_id AS autopilot_workspace_id
|
|
FROM autopilot_trigger t
|
|
JOIN autopilot a ON t.autopilot_id = a.id
|
|
WHERE t.kind = 'schedule'
|
|
AND t.enabled = true
|
|
AND t.next_run_at IS NULL
|
|
AND t.cron_expression IS NOT NULL
|
|
AND a.status = 'active';
|
|
|
|
-- =====================
|
|
-- Failure-rate auto-pause
|
|
-- =====================
|
|
|
|
-- name: SelectAutopilotsExceedingFailureThreshold :many
|
|
-- Find active autopilots whose recent run failure rate exceeds the threshold.
|
|
-- Counts only "real" terminal runs (completed | failed). 'skipped' is
|
|
-- excluded from BOTH numerator and denominator: an admission-skipped run
|
|
-- (e.g. assignee runtime offline at dispatch time, MUL-1899) is neither a
|
|
-- success nor a failure, so it must not dilute the failure ratio (which
|
|
-- would let a 100%-failing autopilot mask itself behind a wall of skips)
|
|
-- nor inflate it. issue_created/running are still excluded so in-flight
|
|
-- work isn't penalised.
|
|
-- Used by the failure monitor to auto-pause sustained-failure autopilots
|
|
-- (the canonical example from MUL-1336 was an autopilot scheduled every 5 min
|
|
-- that 100% failed for days, burning ~1.5k useless tasks per week).
|
|
WITH stats AS (
|
|
SELECT autopilot_id,
|
|
count(*) FILTER (WHERE status IN ('completed', 'failed')) AS total,
|
|
count(*) FILTER (WHERE status = 'failed') AS failed
|
|
FROM autopilot_run
|
|
WHERE created_at >= sqlc.arg('since')::timestamptz
|
|
GROUP BY autopilot_id
|
|
)
|
|
SELECT a.id, a.workspace_id, a.title, a.assignee_id,
|
|
a.created_by_type, a.created_by_id,
|
|
s.total::bigint AS total_runs,
|
|
s.failed::bigint AS failed_runs
|
|
FROM autopilot a
|
|
JOIN stats s ON s.autopilot_id = a.id
|
|
WHERE a.status = 'active'
|
|
AND s.total >= sqlc.arg('min_runs')::bigint
|
|
AND s.failed::float8 / NULLIF(s.total, 0)::float8 >= sqlc.arg('fail_ratio_threshold')::float8
|
|
ORDER BY s.failed DESC, a.id ASC;
|
|
|
|
-- name: SystemPauseAutopilot :one
|
|
-- Atomically pauses an autopilot only if it is currently active. Returns no
|
|
-- rows when the autopilot was already paused/archived (or another worker
|
|
-- raced first), letting the caller treat that as a benign no-op rather than
|
|
-- an error.
|
|
UPDATE autopilot
|
|
SET status = 'paused', updated_at = now()
|
|
WHERE id = $1 AND status = 'active'
|
|
RETURNING *;
|