mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-27 21:33:41 +02:00
* refactor(autopilot): migrate scheduled dispatch to scheduler.Manager
PR 3 of 3 for the scheduled-Autopilot refactor on MUL-3551.
Replaces the legacy cmd/server/autopilot_scheduler.go goroutine
(30 s app-clock polling, app-time cron advancement, weak crash
recovery) with a JobSpec registered on the existing
scheduler.Manager. sys_cron_executions is now the lease + audit
table for scheduled Autopilot occurrences, and the unique key on
(job_name, scope_kind, scope_id, plan_time) is the primary
guarantee that the same planned fire time cannot produce two runs.
What changed
* server/internal/scheduler/jobs_autopilot.go
New AutopilotScheduleDispatchJob factory:
- scope_kind = "autopilot_trigger", scope_id = trigger.id
- PlansForScope hook (from PR 1) enumerates cron occurrences
in (lastPlan, dbNow] and collapses missed fires to the most
recent one (CatchUpLatestOnly — same policy the legacy
goroutine had, now provable via a one-row-per-tick audit).
- Handler re-loads trigger + autopilot inside the handler so a
between-tick state change (paused, disabled, deleted) takes
effect immediately and is recorded as a no-op SUCCESS row
with skipped_reason in the result JSON.
- Calls AutopilotService.DispatchAutopilotForPlan (from PR 2)
for the actual run creation; that path is itself idempotent
on (trigger_id, planned_at), so a stale-steal retry reuses
the run created by the prior attempt instead of duplicating.
- RunTimeout=2m, StaleTimeout=5m, HeartbeatInterval=30s,
AllowStaleReentry=true, MaxAttempts=3, RetryBackoff
[1m, 5m, 15m], MaxPlansPerTick=5 (safety cap).
* server/internal/scheduler/manager.go
Manager.runOnce promoted to RunOnce (exported) so external test
packages can drive deterministic ticks; existing call sites in
this package + cmd/server tests updated.
* server/internal/service/cron.go
NextOccurrenceAfterUTC and NextOccurrencesUTC: cron evaluators
that take an explicit "now" instant. Callers pass dbNow() so
schedule decisions stay consistent across app instances with
clock skew. Legacy ComputeNextRun is preserved (delegating to
NextOccurrenceAfterUTC with time.Now()) for the display-only
autopilot_trigger.next_run_at write path — scheduling decisions
no longer use it.
* server/pkg/db/queries/autopilot.sql
ListSchedulableAutopilotTriggers replaces the legacy
ClaimDueScheduleTriggers (the new path no longer mutates
autopilot_trigger.next_run_at on claim). RecoverLostTriggers
removed — sys_cron_executions lease theft now handles crash
recovery without an in-handler restart sweep.
* server/cmd/server/main.go
The "go runAutopilotScheduler(...)" line is gone. The new
JobSpec is registered alongside TaskUsageHourlyJob on the
existing schedulerMgr (still using sweepCtx for lifecycle).
* server/cmd/server/autopilot_scheduler.go DELETED.
Tests
* server/internal/service/cron_test.go — unit tests for the cron
helpers: timezone-aware enumeration, half-open (after, until]
window, plan_time-exclusive "after", invalid inputs surface
parse errors, and the "ignores wall clock" property the
scheduler relies on.
* server/cmd/server/autopilot_schedule_job_test.go — DB-backed
integration tests:
- DispatchesOnce: one tick → 1 SUCCESS exec row + 1
autopilot_run with planned_at set; a second tick does not
regress the count.
- MissedSchedulesCollapse: an hour of missed */5 fires
produce a single autopilot_run, not 12.
- CrashRecovery: simulated stale RUNNING lease at the same
plan_time → second tick reclaims it and DOES NOT duplicate
autopilot_run.
- TwoRunnersSingleWinner: two concurrent
scheduler.Manager instances on the same trigger →
per-plan_time uniqueness holds (sys_cron_executions never
has two RUNNING rows at the same plan_time, autopilot_run
count == exec row count).
- DisabledTriggerSkips: a trigger disabled between
scope-list and tick produces no exec row.
- PausedAutopilotSkipsAtHandler: an autopilot paused after
the first tick does not produce a new exec row.
- BadCronFailsLoudly: an invalid cron expression never fires
dispatch (parse error surfaces in the plan hook).
Existing autopilot listener / squad / dispatch tests still
pass.
* server/internal/scheduler/plans_for_scope_test.go from PR 1
still passes (RunOnce rename only).
Verification
* go build ./...
* go vet ./...
* go test ./internal/scheduler ./internal/service ./cmd/server
./internal/handler — all green.
Rollback
* Reverting this commit re-introduces the legacy goroutine.
Migration 124 (PR 2) and the scheduler hook (PR 1) stay in
place. Autopilot data on disk is forward- and backward-
compatible: planned_at columns are nullable, the legacy
goroutine never reads planned_at and the new job never reads
autopilot_trigger.next_run_at.
Refs MUL-3551
Co-authored-by: multica-agent <github@multica.ai>
* fix(autopilot): scheduler hook retries FAILED plans + tighten tests
Review fix for #4444 (MUL-3551).
Blocker: hook planner skipped the FAILED-with-retry plan_time
`autopilotPlansForScope` unconditionally set
`after = latest.PlanTime` when `latest.Found`, then enumerated cron
occurrences in the half-open interval `(after, dbNow]`. That
EXCLUDED the FAILED plan_time itself, so `tryClaim`'s
"FAILED-with-retry" branch — which only fires when the planner
returns the same plan_time — never ran. A claim + crash sequence
left the FAILED row stuck at attempt<max_attempts forever and the
scheduled occurrence was lost (MUL-3551 acceptance ③).
Fix: hook now branches on `latest.RetryEligible(now)` BEFORE
computing `after`. When the most recent stored row is FAILED with
attempts remaining and next_retry_at <= dbNow, the hook returns
`[latest.PlanTime]` unchanged. tryClaim's retry-from-FAILED path
fires, attempt increments, the run is retried, and the audit row
reaches SUCCESS at the same plan_time. Mirrors the cadence
planner's `info.RetryEligible(now)` branch in manager.plansForTick.
Tests
* TestAutopilotScheduleJobCrashRecovery rewritten to actually
pin the retry contract instead of just "no duplicate run":
- assert first attempt completes at attempt=1 with a real
task_id linkage (the "complete" snapshot the retry must
reuse);
- simulate a crash mid-dispatch (status=RUNNING, expired
stale_after, ghost lease_token);
- assert tick 2 transitions the SAME exec row (same plan_time)
to status=SUCCESS at attempt=2 (proving the planner did
NOT skip past the FAILED bucket);
- assert autopilot_run stays at exactly one row, reused from
the first attempt — proving DispatchAutopilotForPlan's
complete-run reuse path is what closes the loop.
* TestAutopilotScheduleJobPausedAutopilotSkipsAtHandler rewritten
to invoke `job.Handler` directly (the previous version drove
`mgr.RunOnce` which short-circuited at the scope-list SQL
filter and never reached the handler). The new test pauses the
autopilot AFTER setup, calls the handler with a fabricated
HandlerInput, and asserts the handler returns
skipped_reason=autopilot_inactive without creating an
autopilot_run.
* TestAutopilotScheduleJobBadCronFailsLoudly renamed to
TestAutopilotScheduleJobBadCronStaysSilent and updated to
match the real implementation: a parse error in the plan hook
surfaces as a manager-level warning log, NOT a
sys_cron_executions row (no plan_time was ever claimed). The
test now asserts zero exec rows AND zero autopilot_run rows,
documenting that bad cron is a permanent configuration error
(caught at HTTP create/update time first), not a transient
failure that belongs in the retry envelope.
Refs MUL-3551
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
252 lines
8.7 KiB
Go
252 lines
8.7 KiB
Go
package scheduler
|
|
|
|
import (
|
|
"context"
|
|
"strings"
|
|
"sync/atomic"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
// TestJobSpecValidatePlansForScopeRelaxesCadence covers the relaxed
|
|
// JobSpec.validate() rule: when PlansForScope is set, Cadence is
|
|
// optional because the hook owns plan_time selection (used by the
|
|
// Autopilot schedule job, where each trigger's cron expression is
|
|
// arbitrary and does not fit a single Cadence grid).
|
|
func TestJobSpecValidatePlansForScopeRelaxesCadence(t *testing.T) {
|
|
base := JobSpec{
|
|
Name: "hook_validate",
|
|
RunTimeout: time.Minute,
|
|
StaleTimeout: 2 * time.Minute,
|
|
HeartbeatInterval: 30 * time.Second,
|
|
MaxAttempts: 1,
|
|
Scopes: StaticScopes(ScopeGlobal),
|
|
Handler: func(ctx context.Context, in HandlerInput) (HandlerResult, error) {
|
|
return HandlerResult{}, nil
|
|
},
|
|
}
|
|
|
|
t.Run("cadence still required without hook", func(t *testing.T) {
|
|
j := base
|
|
err := j.validate()
|
|
if err == nil {
|
|
t.Fatalf("expected validate error when both Cadence and PlansForScope are unset")
|
|
}
|
|
if !strings.Contains(err.Error(), "cadence must be > 0") {
|
|
t.Fatalf("expected cadence error, got %v", err)
|
|
}
|
|
})
|
|
|
|
t.Run("cadence optional when hook is set", func(t *testing.T) {
|
|
j := base
|
|
j.PlansForScope = func(ctx context.Context, scope Scope, now time.Time, latest LatestPlanInfo) ([]time.Time, error) {
|
|
return nil, nil
|
|
}
|
|
if err := j.validate(); err != nil {
|
|
t.Fatalf("expected validate to pass with PlansForScope and no Cadence; got %v", err)
|
|
}
|
|
})
|
|
|
|
t.Run("every_plan max_plans_per_tick still required without hook", func(t *testing.T) {
|
|
j := base
|
|
j.Cadence = 5 * time.Minute
|
|
j.CatchUpMode = CatchUpEveryPlan
|
|
err := j.validate()
|
|
if err == nil || !strings.Contains(err.Error(), "max_plans_per_tick") {
|
|
t.Fatalf("expected max_plans_per_tick error for every_plan without hook, got %v", err)
|
|
}
|
|
})
|
|
|
|
t.Run("every_plan max_plans_per_tick optional with hook", func(t *testing.T) {
|
|
j := base
|
|
j.CatchUpMode = CatchUpEveryPlan // legal but ignored when hook is set
|
|
j.PlansForScope = func(ctx context.Context, scope Scope, now time.Time, latest LatestPlanInfo) ([]time.Time, error) {
|
|
return nil, nil
|
|
}
|
|
if err := j.validate(); err != nil {
|
|
t.Fatalf("hook-driven jobs may leave max_plans_per_tick=0; got %v", err)
|
|
}
|
|
})
|
|
|
|
t.Run("other invariants still fire", func(t *testing.T) {
|
|
// RunTimeout > 0 must still be enforced even when the hook is set.
|
|
j := base
|
|
j.PlansForScope = func(ctx context.Context, scope Scope, now time.Time, latest LatestPlanInfo) ([]time.Time, error) {
|
|
return nil, nil
|
|
}
|
|
j.RunTimeout = 0
|
|
err := j.validate()
|
|
if err == nil || !strings.Contains(err.Error(), "run_timeout") {
|
|
t.Fatalf("expected run_timeout invariant to still fire with hook, got %v", err)
|
|
}
|
|
})
|
|
}
|
|
|
|
// TestManagerPlansForScopeHookDrivesPlans verifies end-to-end that a
|
|
// JobSpec.PlansForScope hook fully replaces the Cadence planner:
|
|
//
|
|
// - The hook is invoked with a fresh LatestPlanInfo{Found:false} on
|
|
// first tick, then with the prior plan on the second tick.
|
|
// - Every plan_time the hook returns is claimed and finalised
|
|
// SUCCESS by the manager — proof the hook is wired through the
|
|
// same tryClaim/runClaimed lease path as Cadence-driven jobs.
|
|
// - MaxPlansPerTick caps an over-eager hook without erroring.
|
|
// - Cadence=0 is legal when PlansForScope is set.
|
|
//
|
|
// Lives in the integration tier (requires DATABASE_URL) because it
|
|
// exercises the manager's actual SQL primitives, not a stub.
|
|
func TestManagerPlansForScopeHookDrivesPlans(t *testing.T) {
|
|
pool := integrationPool(t)
|
|
job := newTestJobSpec(uniqueJobName(t, "hook_planner"))
|
|
t.Cleanup(func() { cleanupExecutions(t, pool, job.Name) })
|
|
|
|
ctx := context.Background()
|
|
now, err := dbNow(ctx, pool)
|
|
if err != nil {
|
|
t.Fatalf("dbNow: %v", err)
|
|
}
|
|
|
|
// Hook-driven job: arbitrary plan_times, no Cadence.
|
|
job.Cadence = 0
|
|
job.CatchUpMode = CatchUpLatestOnly // ignored when hook is set, but exercise the path
|
|
job.MaxPlansPerTick = 2 // safety cap; we'll return 3 from the hook below
|
|
|
|
// Plan_times the hook will offer in tick 1. Deliberately NOT on a
|
|
// uniform Cadence grid so the Cadence planner would never produce
|
|
// them — proving the hook bypasses FloorPlan entirely.
|
|
t0 := now.Add(-3 * time.Minute).Truncate(time.Second)
|
|
t1 := now.Add(-2*time.Minute - 17*time.Second).Truncate(time.Second)
|
|
t2 := now.Add(-1 * time.Minute).Truncate(time.Second)
|
|
t3 := now.Add(-15 * time.Second).Truncate(time.Second) // beyond MaxPlansPerTick; should be dropped
|
|
|
|
var hookCalls atomic.Int32
|
|
var lastSeenFound atomic.Bool
|
|
var lastSeenPlan atomic.Pointer[time.Time]
|
|
|
|
job.PlansForScope = func(ctx context.Context, scope Scope, ts time.Time, latest LatestPlanInfo) ([]time.Time, error) {
|
|
hookCalls.Add(1)
|
|
lastSeenFound.Store(latest.Found)
|
|
if latest.Found {
|
|
pt := latest.PlanTime
|
|
lastSeenPlan.Store(&pt)
|
|
} else {
|
|
lastSeenPlan.Store(nil)
|
|
}
|
|
// Realistic hook contract: return only plan_times strictly
|
|
// after the most recent stored one. This is exactly what the
|
|
// Autopilot scheduler will do — compute cron occurrences in
|
|
// (latest.PlanTime, now]. We always *could* return everything
|
|
// and rely on tryClaim's conflict-no-op for idempotency, but
|
|
// returning fewer plans is cheaper and exercises the
|
|
// LatestPlanInfo plumbing.
|
|
all := []time.Time{t0, t1, t2, t3}
|
|
var out []time.Time
|
|
for _, p := range all {
|
|
if !latest.Found || p.After(latest.PlanTime) {
|
|
out = append(out, p)
|
|
}
|
|
}
|
|
return out, nil
|
|
}
|
|
|
|
mgr := NewManager(pool, Options{RunnerID: "hook-planner-runner"})
|
|
if err := mgr.Register(*job); err != nil {
|
|
t.Fatalf("register hook-driven job: %v", err)
|
|
}
|
|
|
|
// Tick 1: no prior plan, hook returns all 4 plans, MaxPlansPerTick
|
|
// truncates to {t0, t1}.
|
|
if err := mgr.RunOnce(ctx); err != nil {
|
|
t.Fatalf("runOnce tick 1: %v", err)
|
|
}
|
|
if hookCalls.Load() == 0 {
|
|
t.Fatalf("hook was never called on tick 1")
|
|
}
|
|
if lastSeenFound.Load() {
|
|
t.Fatalf("first tick should see LatestPlanInfo{Found:false}")
|
|
}
|
|
|
|
rows := dumpJobRows(t, pool, job.Name)
|
|
if len(rows) != 2 {
|
|
t.Fatalf("expected 2 plans claimed (MaxPlansPerTick=2), got %d: %+v", len(rows), rows)
|
|
}
|
|
for _, r := range rows {
|
|
if r.Status != "SUCCESS" {
|
|
t.Fatalf("hook-claimed plan should finish SUCCESS, got %q at %s", r.Status, r.PlanTime)
|
|
}
|
|
}
|
|
wantPlans := []time.Time{t0.UTC(), t1.UTC()}
|
|
for i, r := range rows {
|
|
if !r.PlanTime.Equal(wantPlans[i]) {
|
|
t.Fatalf("plan[%d] = %s; want %s (MaxPlansPerTick should preserve hook order)",
|
|
i, r.PlanTime.Format(time.RFC3339Nano), wantPlans[i].Format(time.RFC3339Nano))
|
|
}
|
|
}
|
|
|
|
// Tick 2: latest stored plan is t1. The hook now returns only the
|
|
// plans strictly after t1 — i.e. {t2, t3}. MaxPlansPerTick=2 lets
|
|
// both through this tick. The hook must see Found=true with the
|
|
// latest plan_time (t1, picked by latestPlan ORDER BY plan_time
|
|
// DESC).
|
|
hookCalls.Store(0)
|
|
if err := mgr.RunOnce(ctx); err != nil {
|
|
t.Fatalf("runOnce tick 2: %v", err)
|
|
}
|
|
if hookCalls.Load() == 0 {
|
|
t.Fatalf("hook was never called on tick 2")
|
|
}
|
|
if !lastSeenFound.Load() {
|
|
t.Fatalf("second tick should see LatestPlanInfo{Found:true}")
|
|
}
|
|
if got := lastSeenPlan.Load(); got == nil || !got.Equal(t1.UTC()) {
|
|
var gotStr string
|
|
if got != nil {
|
|
gotStr = got.Format(time.RFC3339Nano)
|
|
}
|
|
t.Fatalf("second tick's LatestPlanInfo.PlanTime should be the most recent stored plan; want %s got %s",
|
|
t1.UTC().Format(time.RFC3339Nano), gotStr)
|
|
}
|
|
|
|
rows = dumpJobRows(t, pool, job.Name)
|
|
if len(rows) != 4 {
|
|
t.Fatalf("expected 4 plans after tick 2 (tick 1 left {t0,t1}, tick 2 adds {t2,t3}), got %d: %+v",
|
|
len(rows), rows)
|
|
}
|
|
for _, r := range rows {
|
|
if r.Status != "SUCCESS" {
|
|
t.Fatalf("all plans should be SUCCESS after tick 2, got %q at %s", r.Status, r.PlanTime)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestManagerPlansForScopeHookEmptyIsNoOp covers the "nothing due"
|
|
// path: an empty plan slice must not error, must not block subsequent
|
|
// scopes, and must not produce any sys_cron_executions row.
|
|
func TestManagerPlansForScopeHookEmptyIsNoOp(t *testing.T) {
|
|
pool := integrationPool(t)
|
|
job := newTestJobSpec(uniqueJobName(t, "hook_empty"))
|
|
t.Cleanup(func() { cleanupExecutions(t, pool, job.Name) })
|
|
|
|
job.Cadence = 0
|
|
var hookCalls atomic.Int32
|
|
job.PlansForScope = func(ctx context.Context, scope Scope, now time.Time, latest LatestPlanInfo) ([]time.Time, error) {
|
|
hookCalls.Add(1)
|
|
return nil, nil
|
|
}
|
|
|
|
mgr := NewManager(pool, Options{RunnerID: "hook-empty-runner"})
|
|
if err := mgr.Register(*job); err != nil {
|
|
t.Fatalf("register: %v", err)
|
|
}
|
|
if err := mgr.RunOnce(context.Background()); err != nil {
|
|
t.Fatalf("runOnce: %v", err)
|
|
}
|
|
if hookCalls.Load() == 0 {
|
|
t.Fatalf("hook was never called for the registered job")
|
|
}
|
|
rows := dumpJobRows(t, pool, job.Name)
|
|
if len(rows) != 0 {
|
|
t.Fatalf("empty hook output must not create rows; got %d", len(rows))
|
|
}
|
|
}
|