Files
multica/server/internal/scheduler/plans_for_scope_test.go
Multica Eve 131ca80a6c refactor(autopilot): migrate scheduled dispatch to scheduler.Manager (3/3 MUL-3551) (#4444)
* refactor(autopilot): migrate scheduled dispatch to scheduler.Manager

PR 3 of 3 for the scheduled-Autopilot refactor on MUL-3551.

Replaces the legacy cmd/server/autopilot_scheduler.go goroutine
(30 s app-clock polling, app-time cron advancement, weak crash
recovery) with a JobSpec registered on the existing
scheduler.Manager. sys_cron_executions is now the lease + audit
table for scheduled Autopilot occurrences, and the unique key on
(job_name, scope_kind, scope_id, plan_time) is the primary
guarantee that the same planned fire time cannot produce two runs.

What changed

  * server/internal/scheduler/jobs_autopilot.go
    New AutopilotScheduleDispatchJob factory:
      - scope_kind = "autopilot_trigger", scope_id = trigger.id
      - PlansForScope hook (from PR 1) enumerates cron occurrences
        in (lastPlan, dbNow] and collapses missed fires to the most
        recent one (CatchUpLatestOnly — same policy the legacy
        goroutine had, now provable via a one-row-per-tick audit).
      - Handler re-loads trigger + autopilot inside the handler so a
        between-tick state change (paused, disabled, deleted) takes
        effect immediately and is recorded as a no-op SUCCESS row
        with skipped_reason in the result JSON.
      - Calls AutopilotService.DispatchAutopilotForPlan (from PR 2)
        for the actual run creation; that path is itself idempotent
        on (trigger_id, planned_at), so a stale-steal retry reuses
        the run created by the prior attempt instead of duplicating.
      - RunTimeout=2m, StaleTimeout=5m, HeartbeatInterval=30s,
        AllowStaleReentry=true, MaxAttempts=3, RetryBackoff
        [1m, 5m, 15m], MaxPlansPerTick=5 (safety cap).

  * server/internal/scheduler/manager.go
    Manager.runOnce promoted to RunOnce (exported) so external test
    packages can drive deterministic ticks; existing call sites in
    this package + cmd/server tests updated.

  * server/internal/service/cron.go
    NextOccurrenceAfterUTC and NextOccurrencesUTC: cron evaluators
    that take an explicit "now" instant. Callers pass dbNow() so
    schedule decisions stay consistent across app instances with
    clock skew. Legacy ComputeNextRun is preserved (delegating to
    NextOccurrenceAfterUTC with time.Now()) for the display-only
    autopilot_trigger.next_run_at write path — scheduling decisions
    no longer use it.

  * server/pkg/db/queries/autopilot.sql
    ListSchedulableAutopilotTriggers replaces the legacy
    ClaimDueScheduleTriggers (the new path no longer mutates
    autopilot_trigger.next_run_at on claim). RecoverLostTriggers
    removed — sys_cron_executions lease theft now handles crash
    recovery without an in-handler restart sweep.

  * server/cmd/server/main.go
    The "go runAutopilotScheduler(...)" line is gone. The new
    JobSpec is registered alongside TaskUsageHourlyJob on the
    existing schedulerMgr (still using sweepCtx for lifecycle).

  * server/cmd/server/autopilot_scheduler.go DELETED.

Tests

  * server/internal/service/cron_test.go — unit tests for the cron
    helpers: timezone-aware enumeration, half-open (after, until]
    window, plan_time-exclusive "after", invalid inputs surface
    parse errors, and the "ignores wall clock" property the
    scheduler relies on.

  * server/cmd/server/autopilot_schedule_job_test.go — DB-backed
    integration tests:
      - DispatchesOnce: one tick → 1 SUCCESS exec row + 1
        autopilot_run with planned_at set; a second tick does not
        regress the count.
      - MissedSchedulesCollapse: an hour of missed */5 fires
        produce a single autopilot_run, not 12.
      - CrashRecovery: simulated stale RUNNING lease at the same
        plan_time → second tick reclaims it and DOES NOT duplicate
        autopilot_run.
      - TwoRunnersSingleWinner: two concurrent
        scheduler.Manager instances on the same trigger →
        per-plan_time uniqueness holds (sys_cron_executions never
        has two RUNNING rows at the same plan_time, autopilot_run
        count == exec row count).
      - DisabledTriggerSkips: a trigger disabled between
        scope-list and tick produces no exec row.
      - PausedAutopilotSkipsAtHandler: an autopilot paused after
        the first tick does not produce a new exec row.
      - BadCronFailsLoudly: an invalid cron expression never fires
        dispatch (parse error surfaces in the plan hook).
    Existing autopilot listener / squad / dispatch tests still
    pass.

  * server/internal/scheduler/plans_for_scope_test.go from PR 1
    still passes (RunOnce rename only).

Verification

  * go build ./...
  * go vet ./...
  * go test ./internal/scheduler ./internal/service ./cmd/server
    ./internal/handler — all green.

Rollback

  * Reverting this commit re-introduces the legacy goroutine.
    Migration 124 (PR 2) and the scheduler hook (PR 1) stay in
    place. Autopilot data on disk is forward- and backward-
    compatible: planned_at columns are nullable, the legacy
    goroutine never reads planned_at and the new job never reads
    autopilot_trigger.next_run_at.

Refs MUL-3551

Co-authored-by: multica-agent <github@multica.ai>

* fix(autopilot): scheduler hook retries FAILED plans + tighten tests

Review fix for #4444 (MUL-3551).

Blocker: hook planner skipped the FAILED-with-retry plan_time

`autopilotPlansForScope` unconditionally set
`after = latest.PlanTime` when `latest.Found`, then enumerated cron
occurrences in the half-open interval `(after, dbNow]`. That
EXCLUDED the FAILED plan_time itself, so `tryClaim`'s
"FAILED-with-retry" branch — which only fires when the planner
returns the same plan_time — never ran. A claim + crash sequence
left the FAILED row stuck at attempt<max_attempts forever and the
scheduled occurrence was lost (MUL-3551 acceptance ③).

Fix: hook now branches on `latest.RetryEligible(now)` BEFORE
computing `after`. When the most recent stored row is FAILED with
attempts remaining and next_retry_at <= dbNow, the hook returns
`[latest.PlanTime]` unchanged. tryClaim's retry-from-FAILED path
fires, attempt increments, the run is retried, and the audit row
reaches SUCCESS at the same plan_time. Mirrors the cadence
planner's `info.RetryEligible(now)` branch in manager.plansForTick.

Tests

  * TestAutopilotScheduleJobCrashRecovery rewritten to actually
    pin the retry contract instead of just "no duplicate run":
      - assert first attempt completes at attempt=1 with a real
        task_id linkage (the "complete" snapshot the retry must
        reuse);
      - simulate a crash mid-dispatch (status=RUNNING, expired
        stale_after, ghost lease_token);
      - assert tick 2 transitions the SAME exec row (same plan_time)
        to status=SUCCESS at attempt=2 (proving the planner did
        NOT skip past the FAILED bucket);
      - assert autopilot_run stays at exactly one row, reused from
        the first attempt — proving DispatchAutopilotForPlan's
        complete-run reuse path is what closes the loop.

  * TestAutopilotScheduleJobPausedAutopilotSkipsAtHandler rewritten
    to invoke `job.Handler` directly (the previous version drove
    `mgr.RunOnce` which short-circuited at the scope-list SQL
    filter and never reached the handler). The new test pauses the
    autopilot AFTER setup, calls the handler with a fabricated
    HandlerInput, and asserts the handler returns
    skipped_reason=autopilot_inactive without creating an
    autopilot_run.

  * TestAutopilotScheduleJobBadCronFailsLoudly renamed to
    TestAutopilotScheduleJobBadCronStaysSilent and updated to
    match the real implementation: a parse error in the plan hook
    surfaces as a manager-level warning log, NOT a
    sys_cron_executions row (no plan_time was ever claimed). The
    test now asserts zero exec rows AND zero autopilot_run rows,
    documenting that bad cron is a permanent configuration error
    (caught at HTTP create/update time first), not a transient
    failure that belongs in the retry envelope.

Refs MUL-3551

Co-authored-by: multica-agent <github@multica.ai>

---------

Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
2026-06-24 12:09:07 +08:00

252 lines
8.7 KiB
Go

package scheduler
import (
"context"
"strings"
"sync/atomic"
"testing"
"time"
)
// TestJobSpecValidatePlansForScopeRelaxesCadence covers the relaxed
// JobSpec.validate() rule: when PlansForScope is set, Cadence is
// optional because the hook owns plan_time selection (used by the
// Autopilot schedule job, where each trigger's cron expression is
// arbitrary and does not fit a single Cadence grid).
func TestJobSpecValidatePlansForScopeRelaxesCadence(t *testing.T) {
base := JobSpec{
Name: "hook_validate",
RunTimeout: time.Minute,
StaleTimeout: 2 * time.Minute,
HeartbeatInterval: 30 * time.Second,
MaxAttempts: 1,
Scopes: StaticScopes(ScopeGlobal),
Handler: func(ctx context.Context, in HandlerInput) (HandlerResult, error) {
return HandlerResult{}, nil
},
}
t.Run("cadence still required without hook", func(t *testing.T) {
j := base
err := j.validate()
if err == nil {
t.Fatalf("expected validate error when both Cadence and PlansForScope are unset")
}
if !strings.Contains(err.Error(), "cadence must be > 0") {
t.Fatalf("expected cadence error, got %v", err)
}
})
t.Run("cadence optional when hook is set", func(t *testing.T) {
j := base
j.PlansForScope = func(ctx context.Context, scope Scope, now time.Time, latest LatestPlanInfo) ([]time.Time, error) {
return nil, nil
}
if err := j.validate(); err != nil {
t.Fatalf("expected validate to pass with PlansForScope and no Cadence; got %v", err)
}
})
t.Run("every_plan max_plans_per_tick still required without hook", func(t *testing.T) {
j := base
j.Cadence = 5 * time.Minute
j.CatchUpMode = CatchUpEveryPlan
err := j.validate()
if err == nil || !strings.Contains(err.Error(), "max_plans_per_tick") {
t.Fatalf("expected max_plans_per_tick error for every_plan without hook, got %v", err)
}
})
t.Run("every_plan max_plans_per_tick optional with hook", func(t *testing.T) {
j := base
j.CatchUpMode = CatchUpEveryPlan // legal but ignored when hook is set
j.PlansForScope = func(ctx context.Context, scope Scope, now time.Time, latest LatestPlanInfo) ([]time.Time, error) {
return nil, nil
}
if err := j.validate(); err != nil {
t.Fatalf("hook-driven jobs may leave max_plans_per_tick=0; got %v", err)
}
})
t.Run("other invariants still fire", func(t *testing.T) {
// RunTimeout > 0 must still be enforced even when the hook is set.
j := base
j.PlansForScope = func(ctx context.Context, scope Scope, now time.Time, latest LatestPlanInfo) ([]time.Time, error) {
return nil, nil
}
j.RunTimeout = 0
err := j.validate()
if err == nil || !strings.Contains(err.Error(), "run_timeout") {
t.Fatalf("expected run_timeout invariant to still fire with hook, got %v", err)
}
})
}
// TestManagerPlansForScopeHookDrivesPlans verifies end-to-end that a
// JobSpec.PlansForScope hook fully replaces the Cadence planner:
//
// - The hook is invoked with a fresh LatestPlanInfo{Found:false} on
// first tick, then with the prior plan on the second tick.
// - Every plan_time the hook returns is claimed and finalised
// SUCCESS by the manager — proof the hook is wired through the
// same tryClaim/runClaimed lease path as Cadence-driven jobs.
// - MaxPlansPerTick caps an over-eager hook without erroring.
// - Cadence=0 is legal when PlansForScope is set.
//
// Lives in the integration tier (requires DATABASE_URL) because it
// exercises the manager's actual SQL primitives, not a stub.
func TestManagerPlansForScopeHookDrivesPlans(t *testing.T) {
pool := integrationPool(t)
job := newTestJobSpec(uniqueJobName(t, "hook_planner"))
t.Cleanup(func() { cleanupExecutions(t, pool, job.Name) })
ctx := context.Background()
now, err := dbNow(ctx, pool)
if err != nil {
t.Fatalf("dbNow: %v", err)
}
// Hook-driven job: arbitrary plan_times, no Cadence.
job.Cadence = 0
job.CatchUpMode = CatchUpLatestOnly // ignored when hook is set, but exercise the path
job.MaxPlansPerTick = 2 // safety cap; we'll return 3 from the hook below
// Plan_times the hook will offer in tick 1. Deliberately NOT on a
// uniform Cadence grid so the Cadence planner would never produce
// them — proving the hook bypasses FloorPlan entirely.
t0 := now.Add(-3 * time.Minute).Truncate(time.Second)
t1 := now.Add(-2*time.Minute - 17*time.Second).Truncate(time.Second)
t2 := now.Add(-1 * time.Minute).Truncate(time.Second)
t3 := now.Add(-15 * time.Second).Truncate(time.Second) // beyond MaxPlansPerTick; should be dropped
var hookCalls atomic.Int32
var lastSeenFound atomic.Bool
var lastSeenPlan atomic.Pointer[time.Time]
job.PlansForScope = func(ctx context.Context, scope Scope, ts time.Time, latest LatestPlanInfo) ([]time.Time, error) {
hookCalls.Add(1)
lastSeenFound.Store(latest.Found)
if latest.Found {
pt := latest.PlanTime
lastSeenPlan.Store(&pt)
} else {
lastSeenPlan.Store(nil)
}
// Realistic hook contract: return only plan_times strictly
// after the most recent stored one. This is exactly what the
// Autopilot scheduler will do — compute cron occurrences in
// (latest.PlanTime, now]. We always *could* return everything
// and rely on tryClaim's conflict-no-op for idempotency, but
// returning fewer plans is cheaper and exercises the
// LatestPlanInfo plumbing.
all := []time.Time{t0, t1, t2, t3}
var out []time.Time
for _, p := range all {
if !latest.Found || p.After(latest.PlanTime) {
out = append(out, p)
}
}
return out, nil
}
mgr := NewManager(pool, Options{RunnerID: "hook-planner-runner"})
if err := mgr.Register(*job); err != nil {
t.Fatalf("register hook-driven job: %v", err)
}
// Tick 1: no prior plan, hook returns all 4 plans, MaxPlansPerTick
// truncates to {t0, t1}.
if err := mgr.RunOnce(ctx); err != nil {
t.Fatalf("runOnce tick 1: %v", err)
}
if hookCalls.Load() == 0 {
t.Fatalf("hook was never called on tick 1")
}
if lastSeenFound.Load() {
t.Fatalf("first tick should see LatestPlanInfo{Found:false}")
}
rows := dumpJobRows(t, pool, job.Name)
if len(rows) != 2 {
t.Fatalf("expected 2 plans claimed (MaxPlansPerTick=2), got %d: %+v", len(rows), rows)
}
for _, r := range rows {
if r.Status != "SUCCESS" {
t.Fatalf("hook-claimed plan should finish SUCCESS, got %q at %s", r.Status, r.PlanTime)
}
}
wantPlans := []time.Time{t0.UTC(), t1.UTC()}
for i, r := range rows {
if !r.PlanTime.Equal(wantPlans[i]) {
t.Fatalf("plan[%d] = %s; want %s (MaxPlansPerTick should preserve hook order)",
i, r.PlanTime.Format(time.RFC3339Nano), wantPlans[i].Format(time.RFC3339Nano))
}
}
// Tick 2: latest stored plan is t1. The hook now returns only the
// plans strictly after t1 — i.e. {t2, t3}. MaxPlansPerTick=2 lets
// both through this tick. The hook must see Found=true with the
// latest plan_time (t1, picked by latestPlan ORDER BY plan_time
// DESC).
hookCalls.Store(0)
if err := mgr.RunOnce(ctx); err != nil {
t.Fatalf("runOnce tick 2: %v", err)
}
if hookCalls.Load() == 0 {
t.Fatalf("hook was never called on tick 2")
}
if !lastSeenFound.Load() {
t.Fatalf("second tick should see LatestPlanInfo{Found:true}")
}
if got := lastSeenPlan.Load(); got == nil || !got.Equal(t1.UTC()) {
var gotStr string
if got != nil {
gotStr = got.Format(time.RFC3339Nano)
}
t.Fatalf("second tick's LatestPlanInfo.PlanTime should be the most recent stored plan; want %s got %s",
t1.UTC().Format(time.RFC3339Nano), gotStr)
}
rows = dumpJobRows(t, pool, job.Name)
if len(rows) != 4 {
t.Fatalf("expected 4 plans after tick 2 (tick 1 left {t0,t1}, tick 2 adds {t2,t3}), got %d: %+v",
len(rows), rows)
}
for _, r := range rows {
if r.Status != "SUCCESS" {
t.Fatalf("all plans should be SUCCESS after tick 2, got %q at %s", r.Status, r.PlanTime)
}
}
}
// TestManagerPlansForScopeHookEmptyIsNoOp covers the "nothing due"
// path: an empty plan slice must not error, must not block subsequent
// scopes, and must not produce any sys_cron_executions row.
func TestManagerPlansForScopeHookEmptyIsNoOp(t *testing.T) {
pool := integrationPool(t)
job := newTestJobSpec(uniqueJobName(t, "hook_empty"))
t.Cleanup(func() { cleanupExecutions(t, pool, job.Name) })
job.Cadence = 0
var hookCalls atomic.Int32
job.PlansForScope = func(ctx context.Context, scope Scope, now time.Time, latest LatestPlanInfo) ([]time.Time, error) {
hookCalls.Add(1)
return nil, nil
}
mgr := NewManager(pool, Options{RunnerID: "hook-empty-runner"})
if err := mgr.Register(*job); err != nil {
t.Fatalf("register: %v", err)
}
if err := mgr.RunOnce(context.Background()); err != nil {
t.Fatalf("runOnce: %v", err)
}
if hookCalls.Load() == 0 {
t.Fatalf("hook was never called for the registered job")
}
rows := dumpJobRows(t, pool, job.Name)
if len(rows) != 0 {
t.Fatalf("empty hook output must not create rows; got %d", len(rows))
}
}