mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-30 07:10:49 +02:00
* feat(autopilot): skip dispatch when assignee runtime is offline (MUL-1899) Prevents scheduled autopilots from accumulating doomed tasks against offline / archived / unbound agents. Before this change, a paused laptop or crashed daemon would let a 5-minute-cron autopilot pile up thousands of queued agent_task_queue rows that no runtime would ever drain — this is the dominant source of the 89k stuck-task backlog flagged in MUL-1899. DispatchAutopilot now performs a pre-flight admission check on the assignee agent's runtime status. If the runtime is not 'online' (or the agent is archived / has no runtime bound / has no assignee), the run is recorded as 'skipped' with a failure_reason and no task is enqueued. Skipped runs still emit autopilot:run.done so the UI / activity feed reflect that the trigger fired and was evaluated. Skipped runs are deliberately NOT counted toward the failure-ratio auto-pause: a user who closes their laptop overnight should not have their autopilot paused. Sustained server-side failures keep their existing pause path via the failure monitor. Tests: added an integration test that creates an offline runtime and asserts DispatchAutopilot records a skipped run with no task enqueued. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * feat(scheduler): expire stale queued tasks via TTL sweeper (MUL-1899) Companion to the dispatch-time admission gate added in this PR. The admission gate prevents *new* tasks from being enqueued against an offline runtime, but it does not drain the historical backlog (~89k stuck queued rows observed at MUL-1899 baseline) and does not help when a runtime goes offline *after* a task has already been queued. This adds a passive TTL sweeper: - New SQL query `ExpireStaleQueuedTasks` transitions queued tasks older than the TTL to status='failed' with failure_reason='queued_expired' and a clear error message. - Sweep is capped per tick (`queuedExpireBatchSize`, default 500) via a CTE+LIMIT so that draining a large backlog cannot monopolise the DB on a single tick. At 30s ticks the worst case is 60k rows/hour. - Wired into the existing 30s `runRuntimeSweeper` loop alongside `sweepStaleTasks` and reuses `taskSvc.HandleFailedTasks` so the expired tasks broadcast `task:failed` events, reconcile agent status, and roll back any in-progress issues — same lifecycle as any other failed task. - Default TTL = 2h. Conservatively above any reasonable "queued behind a long-running task" window (default agent timeout is 2h, sweeper runs every 30s) so legitimate work isn't expired. - Integration tests cover the happy path (stale → expired, fresh → left alone, correct status/reason/error) and the per-tick batch cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * fix(autopilot): address review blockers from PR #2311 (MUL-1899) GPT-Boy review of the offline-runtime + queued-TTL PR flagged four blockers; this commit addresses them all. 1. Restore the 'skipped' autopilot_run status in the DB constraint. Migration 043 had removed 'skipped' along with the now-defunct concurrency_policy feature, so the new admission gate's INSERT of status='skipped' violated `autopilot_run_status_check` and broke `TestAutopilotDispatchSkipsWhenRuntimeOffline` in CI. New migration 079 re-adds 'skipped' to the CHECK list. The down migration migrates skipped → failed before re-tightening, mirror- ing what 043 did for the original removal. 2. Make `ExpireStaleQueuedTasks` race-safe. The CTE-then-UPDATE pattern could clobber a task that the daemon claimed between victim selection and the outer update. Two guards added: - `FOR UPDATE SKIP LOCKED` in the CTE so we never wait on a row that's currently being claimed (and never block the claim path either). - The outer UPDATE now re-checks `t.status = 'queued'` AND the TTL predicate so even if a row's lock is released after a successful claim, we cannot transition a now-dispatched/ running task to 'failed'. 3. Add a partial index for the queued-TTL sweeper. `idx_agent_task_queue_queued_created_at` on `created_at WHERE status = 'queued'` — keeps the 30s sweep query (status=queued AND created_at < ... ORDER BY created_at LIMIT 500) cheap even when historical terminal rows accumulate (~89k+ at MUL-1899 baseline). The partial predicate keeps the index tiny because only in-flight rows live in 'queued'. 4. Fix the failure-monitor denominator. `SelectAutopilotsExceedingFailureThreshold` had been counting 'skipped' toward total runs, which would have diluted the failure ratio: a 100%-failing autopilot could mask itself behind a wall of admission skips. With 'skipped' restored as a real status, the auto-pause monitor must explicitly exclude it from BOTH numerator and denominator — admission skips are neither a success nor a failure. Verified: `go test ./cmd/server/... ./internal/service/...` passes (including TestAutopilotDispatchSkipsWhenRuntimeOffline, TestExpireStaleQueuedTasks, TestExpireStaleQueuedTasksRespectsBatch Limit). `go build ./... && go vet ./...` clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * fix(migrations): split queued-task TTL index into concurrent migration Per PR #2311 review: agent_task_queue is a hot table, so building the new partial index with plain CREATE INDEX inside migration 079 would hold ACCESS EXCLUSIVE on the queue and block dispatch during deploy. The migration runner does not allow CONCURRENTLY to share a file with other statements (documented in 068), so split the index into its own single-statement file 080 — matching the existing pattern in 035 / 067 / 074 / 075 / 078. Migration 079 keeps the autopilot_run constraint change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Eve <eve@multica-ai.local> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai>
587 lines
18 KiB
Go
587 lines
18 KiB
Go
package service
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"log/slog"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/jackc/pgx/v5"
|
|
"github.com/jackc/pgx/v5/pgtype"
|
|
"github.com/multica-ai/multica/server/internal/analytics"
|
|
"github.com/multica-ai/multica/server/internal/events"
|
|
"github.com/multica-ai/multica/server/internal/util"
|
|
db "github.com/multica-ai/multica/server/pkg/db/generated"
|
|
"github.com/multica-ai/multica/server/pkg/protocol"
|
|
)
|
|
|
|
// TxStarter abstracts transaction creation (satisfied by pgxpool.Pool).
|
|
type TxStarter interface {
|
|
Begin(ctx context.Context) (pgx.Tx, error)
|
|
}
|
|
|
|
type AutopilotService struct {
|
|
Queries *db.Queries
|
|
TxStarter TxStarter
|
|
Bus *events.Bus
|
|
TaskSvc *TaskService
|
|
}
|
|
|
|
func NewAutopilotService(q *db.Queries, tx TxStarter, bus *events.Bus, taskSvc *TaskService) *AutopilotService {
|
|
return &AutopilotService{Queries: q, TxStarter: tx, Bus: bus, TaskSvc: taskSvc}
|
|
}
|
|
|
|
// DispatchAutopilot is the core execution entry point.
|
|
// It creates a run and either creates an issue or enqueues a direct agent task
|
|
// depending on execution_mode.
|
|
//
|
|
// Before any work is queued we run an admission check against the assignee
|
|
// agent's runtime: if it is not online, we record a `skipped` run with a
|
|
// failure_reason and return without enqueueing. This is the "触发时准入" gate
|
|
// from MUL-1899 — without it a paused laptop / offline daemon causes scheduled
|
|
// autopilots to pile thousands of doomed tasks onto agent_task_queue.
|
|
func (s *AutopilotService) DispatchAutopilot(
|
|
ctx context.Context,
|
|
autopilot db.Autopilot,
|
|
triggerID pgtype.UUID,
|
|
source string,
|
|
payload []byte,
|
|
) (*db.AutopilotRun, error) {
|
|
if reason, skip := s.shouldSkipDispatch(ctx, autopilot); skip {
|
|
return s.recordSkippedRun(ctx, autopilot, triggerID, source, payload, reason)
|
|
}
|
|
|
|
// Determine initial status based on execution mode.
|
|
initialStatus := "issue_created"
|
|
if autopilot.ExecutionMode == "run_only" {
|
|
initialStatus = "running"
|
|
}
|
|
|
|
run, err := s.Queries.CreateAutopilotRun(ctx, db.CreateAutopilotRunParams{
|
|
AutopilotID: autopilot.ID,
|
|
TriggerID: triggerID,
|
|
Source: source,
|
|
Status: initialStatus,
|
|
TriggerPayload: payload,
|
|
})
|
|
if err != nil {
|
|
return nil, fmt.Errorf("create run: %w", err)
|
|
}
|
|
s.captureAutopilotRunStarted(autopilot, run, source)
|
|
|
|
switch autopilot.ExecutionMode {
|
|
case "create_issue":
|
|
if err := s.dispatchCreateIssue(ctx, autopilot, &run); err != nil {
|
|
s.failRun(ctx, run.ID, err.Error())
|
|
s.captureAutopilotRunFailed(autopilot, run, source, err.Error())
|
|
return &run, fmt.Errorf("dispatch create_issue: %w", err)
|
|
}
|
|
case "run_only":
|
|
if err := s.dispatchRunOnly(ctx, autopilot, &run); err != nil {
|
|
s.failRun(ctx, run.ID, err.Error())
|
|
s.captureAutopilotRunFailed(autopilot, run, source, err.Error())
|
|
return &run, fmt.Errorf("dispatch run_only: %w", err)
|
|
}
|
|
default:
|
|
s.failRun(ctx, run.ID, "unknown execution_mode: "+autopilot.ExecutionMode)
|
|
s.captureAutopilotRunFailed(autopilot, run, source, "unknown execution_mode: "+autopilot.ExecutionMode)
|
|
return &run, fmt.Errorf("unknown execution_mode: %s", autopilot.ExecutionMode)
|
|
}
|
|
|
|
// Update last_run_at on the autopilot.
|
|
s.Queries.UpdateAutopilotLastRunAt(ctx, autopilot.ID)
|
|
|
|
// Publish run start event.
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventAutopilotRunStart,
|
|
WorkspaceID: util.UUIDToString(autopilot.WorkspaceID),
|
|
ActorType: "system",
|
|
Payload: map[string]any{
|
|
"run_id": util.UUIDToString(run.ID),
|
|
"autopilot_id": util.UUIDToString(autopilot.ID),
|
|
"source": source,
|
|
"status": run.Status,
|
|
},
|
|
})
|
|
|
|
return &run, nil
|
|
}
|
|
|
|
// dispatchCreateIssue creates an issue and enqueues a task for the agent.
|
|
func (s *AutopilotService) dispatchCreateIssue(ctx context.Context, ap db.Autopilot, run *db.AutopilotRun) error {
|
|
tx, err := s.TxStarter.Begin(ctx)
|
|
if err != nil {
|
|
return fmt.Errorf("begin tx: %w", err)
|
|
}
|
|
defer tx.Rollback(ctx)
|
|
|
|
qtx := s.Queries.WithTx(tx)
|
|
|
|
// Get next issue number.
|
|
issueNumber, err := qtx.IncrementIssueCounter(ctx, ap.WorkspaceID)
|
|
if err != nil {
|
|
return fmt.Errorf("increment issue counter: %w", err)
|
|
}
|
|
|
|
title := s.interpolateTemplate(ap)
|
|
description := s.buildIssueDescription(ap)
|
|
|
|
issue, err := qtx.CreateIssueWithOrigin(ctx, db.CreateIssueWithOriginParams{
|
|
WorkspaceID: ap.WorkspaceID,
|
|
Title: title,
|
|
Description: description,
|
|
Status: "todo",
|
|
Priority: "none",
|
|
AssigneeType: pgtype.Text{String: "agent", Valid: true},
|
|
AssigneeID: ap.AssigneeID,
|
|
CreatorType: ap.CreatedByType,
|
|
CreatorID: ap.CreatedByID,
|
|
ParentIssueID: pgtype.UUID{},
|
|
Position: 0,
|
|
DueDate: pgtype.Timestamptz{},
|
|
Number: issueNumber,
|
|
ProjectID: pgtype.UUID{},
|
|
OriginType: pgtype.Text{String: "autopilot", Valid: true},
|
|
OriginID: ap.ID,
|
|
})
|
|
if err != nil {
|
|
return fmt.Errorf("create issue: %w", err)
|
|
}
|
|
|
|
if err := tx.Commit(ctx); err != nil {
|
|
return fmt.Errorf("commit tx: %w", err)
|
|
}
|
|
|
|
// Update run with the linked issue.
|
|
updatedRun, err := s.Queries.UpdateAutopilotRunIssueCreated(ctx, db.UpdateAutopilotRunIssueCreatedParams{
|
|
ID: run.ID,
|
|
IssueID: issue.ID,
|
|
})
|
|
if err != nil {
|
|
return fmt.Errorf("link run to issue: %w", err)
|
|
}
|
|
*run = updatedRun
|
|
|
|
// Publish issue:created so the existing event chain fires
|
|
// (subscriber listeners, activity listeners, notification listeners).
|
|
prefix := s.getIssuePrefix(ap.WorkspaceID)
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventIssueCreated,
|
|
WorkspaceID: util.UUIDToString(ap.WorkspaceID),
|
|
ActorType: ap.CreatedByType,
|
|
ActorID: util.UUIDToString(ap.CreatedByID),
|
|
Payload: map[string]any{
|
|
"issue": issueToMap(issue, prefix),
|
|
},
|
|
})
|
|
s.captureIssueCreatedFromAutopilot(ap, run, issue)
|
|
|
|
// Enqueue agent task via the existing flow.
|
|
if _, err := s.TaskSvc.EnqueueTaskForIssue(ctx, issue); err != nil {
|
|
return fmt.Errorf("enqueue task for issue: %w", err)
|
|
}
|
|
|
|
slog.Info("autopilot dispatched (create_issue)",
|
|
"autopilot_id", util.UUIDToString(ap.ID),
|
|
"issue_id", util.UUIDToString(issue.ID),
|
|
"run_id", util.UUIDToString(run.ID),
|
|
)
|
|
return nil
|
|
}
|
|
|
|
// dispatchRunOnly enqueues a direct agent task without creating an issue.
|
|
func (s *AutopilotService) dispatchRunOnly(ctx context.Context, ap db.Autopilot, run *db.AutopilotRun) error {
|
|
agent, err := s.Queries.GetAgent(ctx, ap.AssigneeID)
|
|
if err != nil {
|
|
return fmt.Errorf("load agent: %w", err)
|
|
}
|
|
if agent.ArchivedAt.Valid {
|
|
return fmt.Errorf("agent is archived")
|
|
}
|
|
if !agent.RuntimeID.Valid {
|
|
return fmt.Errorf("agent has no runtime")
|
|
}
|
|
|
|
task, err := s.Queries.CreateAutopilotTask(ctx, db.CreateAutopilotTaskParams{
|
|
AgentID: ap.AssigneeID,
|
|
RuntimeID: agent.RuntimeID,
|
|
Priority: 0,
|
|
AutopilotRunID: run.ID,
|
|
// Snapshot the autopilot title so task rows self-describe later
|
|
// without joining back to autopilot. Truncated for the same
|
|
// transmission-cost reason as comment-driven summaries.
|
|
TriggerSummary: pgtype.Text{
|
|
String: truncateForSummary(ap.Title, triggerSummaryMaxLen),
|
|
Valid: ap.Title != "",
|
|
},
|
|
})
|
|
if err != nil {
|
|
return fmt.Errorf("create autopilot task: %w", err)
|
|
}
|
|
|
|
// Update run with task reference.
|
|
updatedRun, err := s.Queries.UpdateAutopilotRunRunning(ctx, db.UpdateAutopilotRunRunningParams{
|
|
ID: run.ID,
|
|
TaskID: task.ID,
|
|
})
|
|
if err != nil {
|
|
slog.Warn("failed to update run with task_id", "run_id", util.UUIDToString(run.ID), "error", err)
|
|
} else {
|
|
*run = updatedRun
|
|
}
|
|
|
|
// Drop the empty-claim cache and wake the daemon. dispatchRunOnly
|
|
// inserts the task row directly via Queries.CreateAutopilotTask
|
|
// (bypassing TaskService.Enqueue*), so without this the runtime
|
|
// would not get a wakeup and any cached "empty" verdict would
|
|
// stall the task until the TTL expired.
|
|
s.TaskSvc.NotifyTaskEnqueued(ctx, task)
|
|
|
|
slog.Info("autopilot dispatched (run_only)",
|
|
"autopilot_id", util.UUIDToString(ap.ID),
|
|
"task_id", util.UUIDToString(task.ID),
|
|
"run_id", util.UUIDToString(run.ID),
|
|
)
|
|
return nil
|
|
}
|
|
|
|
// SyncRunFromIssue updates the autopilot run when its linked issue reaches a terminal status.
|
|
func (s *AutopilotService) SyncRunFromIssue(ctx context.Context, issue db.Issue) {
|
|
if !issue.OriginType.Valid || issue.OriginType.String != "autopilot" {
|
|
return
|
|
}
|
|
|
|
run, err := s.Queries.GetAutopilotRunByIssue(ctx, issue.ID)
|
|
if err != nil {
|
|
return // no active run linked to this issue
|
|
}
|
|
autopilot, err := s.Queries.GetAutopilot(ctx, run.AutopilotID)
|
|
if err != nil {
|
|
return
|
|
}
|
|
|
|
wsID := util.UUIDToString(issue.WorkspaceID)
|
|
|
|
switch issue.Status {
|
|
case "done", "in_review":
|
|
updatedRun, err := s.Queries.UpdateAutopilotRunCompleted(ctx, db.UpdateAutopilotRunCompletedParams{
|
|
ID: run.ID,
|
|
})
|
|
if err != nil {
|
|
slog.Warn("failed to complete autopilot run", "run_id", util.UUIDToString(run.ID), "error", err)
|
|
return
|
|
}
|
|
s.captureAutopilotRunCompleted(autopilot, updatedRun)
|
|
s.publishRunDone(wsID, updatedRun, "completed")
|
|
case "cancelled", "blocked":
|
|
reason := "issue " + issue.Status
|
|
updatedRun, err := s.Queries.UpdateAutopilotRunFailed(ctx, db.UpdateAutopilotRunFailedParams{
|
|
ID: run.ID,
|
|
FailureReason: pgtype.Text{String: reason, Valid: true},
|
|
})
|
|
if err != nil {
|
|
slog.Warn("failed to fail autopilot run", "run_id", util.UUIDToString(run.ID), "error", err)
|
|
return
|
|
}
|
|
s.captureAutopilotRunFailed(autopilot, updatedRun, updatedRun.Source, reason)
|
|
s.publishRunDone(wsID, updatedRun, "failed")
|
|
}
|
|
}
|
|
|
|
// SyncRunFromTask updates the autopilot run when a run_only task completes or fails.
|
|
func (s *AutopilotService) SyncRunFromTask(ctx context.Context, task db.AgentTaskQueue) {
|
|
if !task.AutopilotRunID.Valid {
|
|
return
|
|
}
|
|
|
|
run, err := s.Queries.GetAutopilotRun(ctx, task.AutopilotRunID)
|
|
if err != nil {
|
|
return
|
|
}
|
|
|
|
autopilot, err := s.Queries.GetAutopilot(ctx, run.AutopilotID)
|
|
if err != nil {
|
|
return
|
|
}
|
|
wsID := util.UUIDToString(autopilot.WorkspaceID)
|
|
|
|
switch task.Status {
|
|
case "completed":
|
|
updatedRun, err := s.Queries.UpdateAutopilotRunCompleted(ctx, db.UpdateAutopilotRunCompletedParams{
|
|
ID: run.ID,
|
|
Result: task.Result,
|
|
})
|
|
if err != nil {
|
|
slog.Warn("failed to complete autopilot run from task", "run_id", util.UUIDToString(run.ID), "error", err)
|
|
return
|
|
}
|
|
s.captureAutopilotRunCompleted(autopilot, updatedRun)
|
|
s.publishRunDone(wsID, updatedRun, "completed")
|
|
case "failed", "cancelled":
|
|
reason := "task " + task.Status
|
|
if task.Error.Valid {
|
|
reason = task.Error.String
|
|
}
|
|
updatedRun, err := s.Queries.UpdateAutopilotRunFailed(ctx, db.UpdateAutopilotRunFailedParams{
|
|
ID: run.ID,
|
|
FailureReason: pgtype.Text{String: reason, Valid: true},
|
|
})
|
|
if err != nil {
|
|
slog.Warn("failed to fail autopilot run from task", "run_id", util.UUIDToString(run.ID), "error", err)
|
|
return
|
|
}
|
|
s.captureAutopilotRunFailed(autopilot, updatedRun, updatedRun.Source, reason)
|
|
s.publishRunDone(wsID, updatedRun, "failed")
|
|
}
|
|
}
|
|
|
|
func (s *AutopilotService) failRun(ctx context.Context, runID pgtype.UUID, reason string) {
|
|
if _, err := s.Queries.UpdateAutopilotRunFailed(ctx, db.UpdateAutopilotRunFailedParams{
|
|
ID: runID,
|
|
FailureReason: pgtype.Text{String: reason, Valid: true},
|
|
}); err != nil {
|
|
slog.Warn("failed to mark autopilot run as failed", "run_id", util.UUIDToString(runID), "error", err)
|
|
}
|
|
}
|
|
|
|
// shouldSkipDispatch is the pre-flight admission check from MUL-1899.
|
|
// Returns (reason, true) when dispatching now would only enqueue a doomed
|
|
// task — i.e. the assignee agent is gone, archived, has no runtime bound, or
|
|
// its runtime is not currently online. Returns ("", false) on the happy path.
|
|
//
|
|
// Errors loading the agent / runtime are logged but treated as "do not skip"
|
|
// so a transient DB hiccup never silently swallows a scheduled run.
|
|
func (s *AutopilotService) shouldSkipDispatch(ctx context.Context, ap db.Autopilot) (string, bool) {
|
|
if !ap.AssigneeID.Valid {
|
|
return "autopilot has no assignee", true
|
|
}
|
|
agent, err := s.Queries.GetAgent(ctx, ap.AssigneeID)
|
|
if err != nil {
|
|
slog.Warn("autopilot admission: failed to load assignee agent",
|
|
"autopilot_id", util.UUIDToString(ap.ID),
|
|
"agent_id", util.UUIDToString(ap.AssigneeID),
|
|
"error", err,
|
|
)
|
|
return "", false
|
|
}
|
|
if agent.ArchivedAt.Valid {
|
|
return "assignee agent is archived", true
|
|
}
|
|
if !agent.RuntimeID.Valid {
|
|
return "assignee agent has no runtime bound", true
|
|
}
|
|
rt, err := s.Queries.GetAgentRuntime(ctx, agent.RuntimeID)
|
|
if err != nil {
|
|
slog.Warn("autopilot admission: failed to load runtime",
|
|
"autopilot_id", util.UUIDToString(ap.ID),
|
|
"runtime_id", util.UUIDToString(agent.RuntimeID),
|
|
"error", err,
|
|
)
|
|
return "", false
|
|
}
|
|
if rt.Status != "online" {
|
|
return "agent runtime is " + rt.Status + " at dispatch time", true
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
// recordSkippedRun persists a `skipped` autopilot_run with the given reason
|
|
// and emits the same WS / analytics signals that a normal terminal transition
|
|
// would. Returns the run + nil error so callers (scheduler tick, manual
|
|
// trigger handler) treat this as a successful — but no-op — dispatch.
|
|
func (s *AutopilotService) recordSkippedRun(
|
|
ctx context.Context,
|
|
autopilot db.Autopilot,
|
|
triggerID pgtype.UUID,
|
|
source string,
|
|
payload []byte,
|
|
reason string,
|
|
) (*db.AutopilotRun, error) {
|
|
run, err := s.Queries.CreateAutopilotRun(ctx, db.CreateAutopilotRunParams{
|
|
AutopilotID: autopilot.ID,
|
|
TriggerID: triggerID,
|
|
Source: source,
|
|
Status: "skipped",
|
|
TriggerPayload: payload,
|
|
})
|
|
if err != nil {
|
|
return nil, fmt.Errorf("create skipped run: %w", err)
|
|
}
|
|
|
|
updated, err := s.Queries.UpdateAutopilotRunSkipped(ctx, db.UpdateAutopilotRunSkippedParams{
|
|
ID: run.ID,
|
|
FailureReason: pgtype.Text{String: reason, Valid: true},
|
|
})
|
|
if err == nil {
|
|
run = updated
|
|
} else {
|
|
slog.Warn("failed to set skip reason on autopilot run",
|
|
"run_id", util.UUIDToString(run.ID), "error", err)
|
|
}
|
|
|
|
slog.Info("autopilot dispatch skipped",
|
|
"autopilot_id", util.UUIDToString(autopilot.ID),
|
|
"run_id", util.UUIDToString(run.ID),
|
|
"source", source,
|
|
"reason", reason,
|
|
)
|
|
|
|
// Bump last_run_at so scheduler advancement and "last seen" UI both
|
|
// reflect that we did evaluate the trigger this tick.
|
|
s.Queries.UpdateAutopilotLastRunAt(ctx, autopilot.ID)
|
|
|
|
s.publishRunDone(util.UUIDToString(autopilot.WorkspaceID), run, "skipped")
|
|
return &run, nil
|
|
}
|
|
|
|
func (s *AutopilotService) publishRunDone(workspaceID string, run db.AutopilotRun, status string) {
|
|
s.Bus.Publish(events.Event{
|
|
Type: protocol.EventAutopilotRunDone,
|
|
WorkspaceID: workspaceID,
|
|
ActorType: "system",
|
|
Payload: map[string]any{
|
|
"run_id": util.UUIDToString(run.ID),
|
|
"autopilot_id": util.UUIDToString(run.AutopilotID),
|
|
"status": status,
|
|
},
|
|
})
|
|
}
|
|
|
|
func (s *AutopilotService) captureIssueCreatedFromAutopilot(ap db.Autopilot, run *db.AutopilotRun, issue db.Issue) {
|
|
if s.TaskSvc == nil || s.TaskSvc.Analytics == nil {
|
|
return
|
|
}
|
|
s.TaskSvc.Analytics.Capture(analytics.IssueCreated(
|
|
autopilotActorID(ap),
|
|
util.UUIDToString(ap.WorkspaceID),
|
|
util.UUIDToString(issue.ID),
|
|
util.UUIDToString(ap.AssigneeID),
|
|
"",
|
|
util.UUIDToString(run.ID),
|
|
analytics.SourceAutopilot,
|
|
))
|
|
}
|
|
|
|
func (s *AutopilotService) captureAutopilotRunStarted(ap db.Autopilot, run db.AutopilotRun, triggerSource string) {
|
|
if s.TaskSvc == nil || s.TaskSvc.Analytics == nil {
|
|
return
|
|
}
|
|
s.TaskSvc.Analytics.Capture(analytics.AutopilotRunStarted(
|
|
autopilotActorID(ap),
|
|
util.UUIDToString(ap.WorkspaceID),
|
|
util.UUIDToString(ap.ID),
|
|
util.UUIDToString(run.ID),
|
|
util.UUIDToString(ap.AssigneeID),
|
|
triggerSource,
|
|
))
|
|
}
|
|
|
|
func (s *AutopilotService) captureAutopilotRunCompleted(ap db.Autopilot, run db.AutopilotRun) {
|
|
if s.TaskSvc == nil || s.TaskSvc.Analytics == nil {
|
|
return
|
|
}
|
|
s.TaskSvc.Analytics.Capture(analytics.AutopilotRunCompleted(
|
|
autopilotActorID(ap),
|
|
util.UUIDToString(ap.WorkspaceID),
|
|
util.UUIDToString(ap.ID),
|
|
util.UUIDToString(run.ID),
|
|
util.UUIDToString(ap.AssigneeID),
|
|
run.Source,
|
|
autopilotRunDurationMS(run),
|
|
))
|
|
}
|
|
|
|
func (s *AutopilotService) captureAutopilotRunFailed(ap db.Autopilot, run db.AutopilotRun, triggerSource, reason string) {
|
|
if s.TaskSvc == nil || s.TaskSvc.Analytics == nil {
|
|
return
|
|
}
|
|
if reason == "" {
|
|
reason = "unknown"
|
|
}
|
|
s.TaskSvc.Analytics.Capture(analytics.AutopilotRunFailed(
|
|
autopilotActorID(ap),
|
|
util.UUIDToString(ap.WorkspaceID),
|
|
util.UUIDToString(ap.ID),
|
|
util.UUIDToString(run.ID),
|
|
util.UUIDToString(ap.AssigneeID),
|
|
triggerSource,
|
|
reason,
|
|
autopilotErrorType(reason),
|
|
false,
|
|
autopilotRunDurationMS(run),
|
|
))
|
|
}
|
|
|
|
func autopilotErrorType(reason string) string {
|
|
switch {
|
|
case strings.Contains(reason, "unknown execution_mode"):
|
|
return "configuration"
|
|
case strings.HasPrefix(reason, "issue "):
|
|
return "issue_terminal"
|
|
case strings.Contains(reason, "create issue"), strings.Contains(reason, "enqueue task"), strings.Contains(reason, "dispatch"):
|
|
return "dispatch_error"
|
|
case strings.HasPrefix(reason, "task "):
|
|
return "task_error"
|
|
default:
|
|
return "autopilot_error"
|
|
}
|
|
}
|
|
|
|
func autopilotActorID(ap db.Autopilot) string {
|
|
id := util.UUIDToString(ap.CreatedByID)
|
|
if ap.CreatedByType == "agent" && id != "" {
|
|
return "agent:" + id
|
|
}
|
|
if id != "" {
|
|
return id
|
|
}
|
|
return "system"
|
|
}
|
|
|
|
func autopilotRunDurationMS(run db.AutopilotRun) int64 {
|
|
if !run.CompletedAt.Valid {
|
|
return 0
|
|
}
|
|
start := run.TriggeredAt
|
|
if !start.Valid {
|
|
start = run.CreatedAt
|
|
}
|
|
if !start.Valid {
|
|
return 0
|
|
}
|
|
ms := run.CompletedAt.Time.Sub(start.Time).Milliseconds()
|
|
if ms < 0 {
|
|
return 0
|
|
}
|
|
return ms
|
|
}
|
|
|
|
// buildIssueDescription appends an autopilot system instruction to the
|
|
// user-provided description, asking the agent to rename the issue after
|
|
// it understands the actual work.
|
|
func (s *AutopilotService) buildIssueDescription(ap db.Autopilot) pgtype.Text {
|
|
now := time.Now().UTC().Format("2006-01-02 15:04 UTC")
|
|
note := fmt.Sprintf("\n\n---\n*Autopilot run triggered at %s. After starting work, rename this issue to accurately reflect what you are doing.*", now)
|
|
base := ap.Description.String
|
|
return pgtype.Text{String: base + note, Valid: true}
|
|
}
|
|
|
|
// interpolateTemplate replaces {{date}} in the issue title template.
|
|
func (s *AutopilotService) interpolateTemplate(ap db.Autopilot) string {
|
|
tmpl := ap.Title
|
|
if ap.IssueTitleTemplate.Valid && ap.IssueTitleTemplate.String != "" {
|
|
tmpl = ap.IssueTitleTemplate.String
|
|
}
|
|
now := time.Now().UTC().Format("2006-01-02")
|
|
return strings.ReplaceAll(tmpl, "{{date}}", now)
|
|
}
|
|
|
|
func (s *AutopilotService) getIssuePrefix(workspaceID pgtype.UUID) string {
|
|
ws, err := s.Queries.GetWorkspace(context.Background(), workspaceID)
|
|
if err != nil {
|
|
return ""
|
|
}
|
|
return ws.IssuePrefix
|
|
}
|