* fix(task): auto-retry transient provider stream cut like chat (MUL-4910)
Claude Code's "API Error: Connection closed mid-response" is a transient
network cut. In unattended issue runs it fell through to
agent_error.unknown / process_failure — neither in retryableReasons — so
the task terminated. Interactive chat only appeared resilient because the
CLI's own in-process retry usually recovers first; there is no
chat-specific retry in Multica. Both paths share the same
finalizeStreamResult -> Classify -> retryableReasons pipeline.
- classify: route "connection closed" / "mid-response" to
agent_error.provider_network, before the exit-status rule so the
"exited with error: exit status N" variant also lands here.
- retry: add provider_network to retryableReasons. It is resume-safe, so
the retry child inherits the session and continues the truncated
conversation instead of restarting.
Tests cover the new classification (incl. exit-status variant) and the
retry/resume flags. Note: mirror the substrings into the MUL-1949 offline
backfill SQL to keep in-flight and historical taxonomies aligned.
Co-authored-by: multica-agent <github@multica.ai>
* feat(task): defer provider_network's final retry ~5s — three-tier (MUL-4910)
Follow-up to the immediate-retry fix: make the connection-closed retry a
three-tier schedule — first run + immediate retry + one retry deferred ~5s —
so a blip that survives the immediate retry gets a short cooldown before the
final attempt instead of firing back-to-back.
Reuses the existing deferred/fire_at primitive (the comment-routing escalation
mechanism) rather than adding new infrastructure: CreateRetryTask gains an
optional fire_at; when set, the child is inserted 'deferred' and the existing
PromoteDueDeferredTasksForRuntime sweeper — already run promote-first on every
claim poll — flips it to 'queued' at fire_at. No migration, no claim-query
change, no daemon change.
- retryAttemptCeiling: raise provider_network's ceiling to 3 (other reasons
keep max_attempts=2); applied in both retryEligible and
MaybeRetryFailedTask's budget pre-check so the primary and sweeper paths agree.
- retryDelayForAttempt: only provider_network's final attempt is deferred (5s);
every other retry — including provider_network's first — stays immediate.
- FailTask + MaybeRetryFailedTask pass fire_at and skip the queued
broadcast/notify for a deferred child (promotion emits them at fire time).
Timing: the deferred child fires on the first claim poll at/after fire_at, so
>= 5s; on an otherwise-idle runtime it can stretch to the poll interval — the
same behaviour deferred escalations already have.
Tests: pure schedule/eligibility coverage (TestProviderNetworkRetrySchedule)
plus a DB test asserting fire_at controls deferred-vs-queued, attempt=3, and
resume-safety.
Co-authored-by: multica-agent <github@multica.ai>
* fix(task): persist reason-aware retry budget; respect max_attempts=1 disable (MUL-4910)
Addresses the pre-merge review must-fix: retryAttemptCeiling raised
provider_network to 3 unconditionally, which (1) overrode the
max_attempts=1 "auto-retry disabled" contract and (2) persisted a
self-contradictory child (attempt=3, max_attempts=2) that leaks to the
task API, so a naive attempt < max_attempts consumer would misjudge the
budget.
- retryAttemptCeiling now returns taskMaxAttempts unchanged when it is <= 1
(disabled stays disabled) and only ever WIDENS otherwise (max(col, 3) for
provider_network) — a higher configured budget is kept.
- CreateRetryTask takes an optional max_attempts; FailTask and
MaybeRetryFailedTask write the reason-aware effective ceiling into the
child so the whole retry chain self-describes (attempt=3, max_attempts=3).
NULL inherits the parent column, so non-provider_network reasons are
unchanged.
Tests:
- pure: ceiling widens to 3, keeps a higher budget, and never revives a
disabled (max_attempts=1) task; eligibility rejects the disabled case.
- DB (end-to-end FailTask): default budget → deferred final child at
attempt=3/max_attempts=3; first failure → immediate child; max_attempts=1
→ no child. Plus CreateRetryTask persists the passed budget / inherits on NULL.
Co-authored-by: multica-agent <github@multica.ai>
---------
Co-authored-by: Bohan-J <bohan@devv.ai>
Co-authored-by: multica-agent <github@multica.ai>
Lift MUL-1949's offline backfill failure_reason taxonomy into a shared
in-flight classifier so the agent_task_queue.failure_reason column is
written with refined values (provider_auth_or_access, context_overflow,
provider_capacity_or_rate_limit, …) at write time rather than waiting on
SQL backfill to re-classify after the fact. PR1 of the Grafana board
plan in MUL-2328 — the upcoming PR2 reuses pkg/taskfailure.AllReasons()
to pre-warm the Prometheus failure_reason label set.
* server/pkg/taskfailure: new package with the canonical 21 Reason
constants (7 platform-side + 14 agent_error.* sub-reasons),
AllReasons() returning a defensive copy, IsAgentError() prefix check,
and Classify(rawError) Reason mirroring the SQL CASE rules from
MUL-1949 (db-boy's analysis). 100% statement coverage.
* server/internal/daemon/daemon.go: route the 'agent_error' coarse
fallback paths (StartTask error, runTask early-return error, CompleteTask
permanent rejection, reportTaskResult default branch) and the
executeAndDrain default error case (chained after classifyPoisonedError)
through taskfailure.Classify so blocked / timeout / unknown-status
results all carry a refined reason on the wire.
* server/internal/service/task.go: FailTask classifies errMsg when the
daemon-supplied failureReason is empty, eliminating the legacy
COALESCE(.., 'agent_error') landing.
* server/internal/daemon/poisoned.go: alias FailureReasonIterationLimit
and FailureReasonAPIInvalidRequest to the canonical taskfailure
constants. agent_fallback_message and codex_semantic_inactivity are
pre-existing operational reasons not in the canonical 21 — kept as
literals for now and revisited in a follow-up PR.
Backfill SQL from MUL-1949 stays as the authoritative offline source of
truth; this PR keeps the in-flight classifier in lock-step with the SQL
CASE expression so historical and future rows share the same taxonomy.
No behavior change for the platform-side reasons (queued_expired,
runtime_offline, runtime_recovery, timeout, etc.) which already align
with the canonical set.
Co-authored-by: Eve <eve@multica-ai.local>
Co-authored-by: multica-agent <github@multica.ai>
