mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-15 22:29:04 +02:00
131ca80a6c
* refactor(autopilot): migrate scheduled dispatch to scheduler.Manager
PR 3 of 3 for the scheduled-Autopilot refactor on MUL-3551.
Replaces the legacy cmd/server/autopilot_scheduler.go goroutine
(30 s app-clock polling, app-time cron advancement, weak crash
recovery) with a JobSpec registered on the existing
scheduler.Manager. sys_cron_executions is now the lease + audit
table for scheduled Autopilot occurrences, and the unique key on
(job_name, scope_kind, scope_id, plan_time) is the primary
guarantee that the same planned fire time cannot produce two runs.
What changed
* server/internal/scheduler/jobs_autopilot.go
New AutopilotScheduleDispatchJob factory:
- scope_kind = "autopilot_trigger", scope_id = trigger.id
- PlansForScope hook (from PR 1) enumerates cron occurrences
in (lastPlan, dbNow] and collapses missed fires to the most
recent one (CatchUpLatestOnly — same policy the legacy
goroutine had, now provable via a one-row-per-tick audit).
- Handler re-loads trigger + autopilot inside the handler so a
between-tick state change (paused, disabled, deleted) takes
effect immediately and is recorded as a no-op SUCCESS row
with skipped_reason in the result JSON.
- Calls AutopilotService.DispatchAutopilotForPlan (from PR 2)
for the actual run creation; that path is itself idempotent
on (trigger_id, planned_at), so a stale-steal retry reuses
the run created by the prior attempt instead of duplicating.
- RunTimeout=2m, StaleTimeout=5m, HeartbeatInterval=30s,
AllowStaleReentry=true, MaxAttempts=3, RetryBackoff
[1m, 5m, 15m], MaxPlansPerTick=5 (safety cap).
* server/internal/scheduler/manager.go
Manager.runOnce promoted to RunOnce (exported) so external test
packages can drive deterministic ticks; existing call sites in
this package + cmd/server tests updated.
* server/internal/service/cron.go
NextOccurrenceAfterUTC and NextOccurrencesUTC: cron evaluators
that take an explicit "now" instant. Callers pass dbNow() so
schedule decisions stay consistent across app instances with
clock skew. Legacy ComputeNextRun is preserved (delegating to
NextOccurrenceAfterUTC with time.Now()) for the display-only
autopilot_trigger.next_run_at write path — scheduling decisions
no longer use it.
* server/pkg/db/queries/autopilot.sql
ListSchedulableAutopilotTriggers replaces the legacy
ClaimDueScheduleTriggers (the new path no longer mutates
autopilot_trigger.next_run_at on claim). RecoverLostTriggers
removed — sys_cron_executions lease theft now handles crash
recovery without an in-handler restart sweep.
* server/cmd/server/main.go
The "go runAutopilotScheduler(...)" line is gone. The new
JobSpec is registered alongside TaskUsageHourlyJob on the
existing schedulerMgr (still using sweepCtx for lifecycle).
* server/cmd/server/autopilot_scheduler.go DELETED.
Tests
* server/internal/service/cron_test.go — unit tests for the cron
helpers: timezone-aware enumeration, half-open (after, until]
window, plan_time-exclusive "after", invalid inputs surface
parse errors, and the "ignores wall clock" property the
scheduler relies on.
* server/cmd/server/autopilot_schedule_job_test.go — DB-backed
integration tests:
- DispatchesOnce: one tick → 1 SUCCESS exec row + 1
autopilot_run with planned_at set; a second tick does not
regress the count.
- MissedSchedulesCollapse: an hour of missed */5 fires
produce a single autopilot_run, not 12.
- CrashRecovery: simulated stale RUNNING lease at the same
plan_time → second tick reclaims it and DOES NOT duplicate
autopilot_run.
- TwoRunnersSingleWinner: two concurrent
scheduler.Manager instances on the same trigger →
per-plan_time uniqueness holds (sys_cron_executions never
has two RUNNING rows at the same plan_time, autopilot_run
count == exec row count).
- DisabledTriggerSkips: a trigger disabled between
scope-list and tick produces no exec row.
- PausedAutopilotSkipsAtHandler: an autopilot paused after
the first tick does not produce a new exec row.
- BadCronFailsLoudly: an invalid cron expression never fires
dispatch (parse error surfaces in the plan hook).
Existing autopilot listener / squad / dispatch tests still
pass.
* server/internal/scheduler/plans_for_scope_test.go from PR 1
still passes (RunOnce rename only).
Verification
* go build ./...
* go vet ./...
* go test ./internal/scheduler ./internal/service ./cmd/server
./internal/handler — all green.
Rollback
* Reverting this commit re-introduces the legacy goroutine.
Migration 124 (PR 2) and the scheduler hook (PR 1) stay in
place. Autopilot data on disk is forward- and backward-
compatible: planned_at columns are nullable, the legacy
goroutine never reads planned_at and the new job never reads
autopilot_trigger.next_run_at.
Refs MUL-3551
Co-authored-by: multica-agent <github@multica.ai>
* fix(autopilot): scheduler hook retries FAILED plans + tighten tests
Review fix for #4444 (MUL-3551).
Blocker: hook planner skipped the FAILED-with-retry plan_time
`autopilotPlansForScope` unconditionally set
`after = latest.PlanTime` when `latest.Found`, then enumerated cron
occurrences in the half-open interval `(after, dbNow]`. That
EXCLUDED the FAILED plan_time itself, so `tryClaim`'s
"FAILED-with-retry" branch — which only fires when the planner
returns the same plan_time — never ran. A claim + crash sequence
left the FAILED row stuck at attempt<max_attempts forever and the
scheduled occurrence was lost (MUL-3551 acceptance ③).
Fix: hook now branches on `latest.RetryEligible(now)` BEFORE
computing `after`. When the most recent stored row is FAILED with
attempts remaining and next_retry_at <= dbNow, the hook returns
`[latest.PlanTime]` unchanged. tryClaim's retry-from-FAILED path
fires, attempt increments, the run is retried, and the audit row
reaches SUCCESS at the same plan_time. Mirrors the cadence
planner's `info.RetryEligible(now)` branch in manager.plansForTick.
Tests
* TestAutopilotScheduleJobCrashRecovery rewritten to actually
pin the retry contract instead of just "no duplicate run":
- assert first attempt completes at attempt=1 with a real
task_id linkage (the "complete" snapshot the retry must
reuse);
- simulate a crash mid-dispatch (status=RUNNING, expired
stale_after, ghost lease_token);
- assert tick 2 transitions the SAME exec row (same plan_time)
to status=SUCCESS at attempt=2 (proving the planner did
NOT skip past the FAILED bucket);
- assert autopilot_run stays at exactly one row, reused from
the first attempt — proving DispatchAutopilotForPlan's
complete-run reuse path is what closes the loop.
* TestAutopilotScheduleJobPausedAutopilotSkipsAtHandler rewritten
to invoke `job.Handler` directly (the previous version drove
`mgr.RunOnce` which short-circuited at the scope-list SQL
filter and never reached the handler). The new test pauses the
autopilot AFTER setup, calls the handler with a fabricated
HandlerInput, and asserts the handler returns
skipped_reason=autopilot_inactive without creating an
autopilot_run.
* TestAutopilotScheduleJobBadCronFailsLoudly renamed to
TestAutopilotScheduleJobBadCronStaysSilent and updated to
match the real implementation: a parse error in the plan hook
surfaces as a manager-level warning log, NOT a
sys_cron_executions row (no plan_time was ever claimed). The
test now asserts zero exec rows AND zero autopilot_run rows,
documenting that bad cron is a permanent configuration error
(caught at HTTP create/update time first), not a transient
failure that belongs in the retry envelope.
Refs MUL-3551
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>