4 Commits

Author	SHA1	Message	Date
Bohan Jiang	c271f80999	MUL-5370 fix: label stalled skill-bundle downloads, align failure-reason copy with the backend taxonomy (#6001) ... * fix(daemon): label stalled skill-bundle downloads and make them retryable A skill bundle that could not be downloaded during task preparation surfaced as the bare string "resolve skill bundles: context deadline exceeded". taskfailure.Classify has no rule for a Go context deadline, so it landed in agent_error.unknown — a bucket that is NOT on the server's retry allowlist. A transient stall therefore became a terminal chat failure carrying a label nobody could act on, and the failure was invisible on the Usage page's Errors breakdown. (MUL-5370) - Add the platform-side reason skill_bundle_unavailable and put it on retryableReasons. Retrying is cheap and safe: the agent process never started, and bundles that did arrive are already cached on disk, so successive attempts converge. - Carry a sentinel error from the resolve loop so the reason is derived structurally rather than by matching the wrapped transport error's text, and name the skill, its declared size and the elapsed wait in the wrap — enough to tell "this bundle is too big for the link" from "the link is dead" without reading daemon logs. - Normalise the wire shape an OLD daemon produces (a non-empty catchall plus the previous "resolve skill bundles:" wrapper) on the server side. Installed daemons upgrade on their own cadence, and FailTask only classifies when the caller supplied nothing, so without this the fix would reach only hosts that happened to update — while the un-upgraded hosts most likely to be hitting the bug kept failing terminally. - Teach Classify about "deadline exceeded" and net/http's "Client.Timeout exceeded while awaiting" so any other Go-side deadline that reaches it as text stops falling into the unknown bucket too. - Backfill historical rows in both agent_task_queue and chat_message. Scoped to agent_error.unknown alone — the old wrapper string postdates the in-flight classifier by three weeks, so no row carrying it can hold the legacy coarse value — which keeps the down migration an exact inverse. Co-authored-by: multica-agent <github@multica.ai> * fix(chat): give chat its own failure copy for the refined reasons #5991 rebuilt the operator-facing failure labels around an open wire string with a raw-value fallback, but the chat bubble kept its own exact-key lookup against the six coarse values from migration 055. So all 14 agent_error.* values still missed and rendered the generic "Something went wrong and the agent couldn't finish replying" — the classification the backend had already computed was discarded at the last step, and that is the message the MUL-5370 reporter saw. - Add resolveFailureReasonKey in packages/core: exact match, else degrade an `agent_error.*` value to its family, else undefined. A reason newer than the shipped client now lands on the family line instead of the fallback. - Rekey the chat copy map by wire value and route it through the helper. Chat deliberately degrades to friendly copy rather than adopting the operator surfaces' raw-value fallback: it is read by the person who just sent a message, and the raw error is one click away under the collapsible. - Add refined chat copy (en / zh-Hans / ja / ko) only where it can say something the family line can't — a different next step: network, auth, quota, rate limit, context overflow, missing/outdated CLI, skill download. - Give skill_bundle_unavailable a label on the web and mobile surfaces and a class on the Usage page's Errors breakdown (runtime — the operator response is "check the daemon's link to Multica", the provider is not involved). - Mobile's two label maps were still coarse-only for the same reason; rekey them by wire value and fill in the refined taxonomy. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Bohan-J <bohan@devv.ai> Co-authored-by: multica-agent <github@multica.ai>	2026-07-28 13:31:29 +08:00
Bohan Jiang	6dba74c3c1	fix(task): auto-retry transient "Connection closed mid-response" like chat (MUL-4910) (#5565) ... * fix(task): auto-retry transient provider stream cut like chat (MUL-4910) Claude Code's "API Error: Connection closed mid-response" is a transient network cut. In unattended issue runs it fell through to agent_error.unknown / process_failure — neither in retryableReasons — so the task terminated. Interactive chat only appeared resilient because the CLI's own in-process retry usually recovers first; there is no chat-specific retry in Multica. Both paths share the same finalizeStreamResult -> Classify -> retryableReasons pipeline. - classify: route "connection closed" / "mid-response" to agent_error.provider_network, before the exit-status rule so the "exited with error: exit status N" variant also lands here. - retry: add provider_network to retryableReasons. It is resume-safe, so the retry child inherits the session and continues the truncated conversation instead of restarting. Tests cover the new classification (incl. exit-status variant) and the retry/resume flags. Note: mirror the substrings into the MUL-1949 offline backfill SQL to keep in-flight and historical taxonomies aligned. Co-authored-by: multica-agent <github@multica.ai> * feat(task): defer provider_network's final retry ~5s — three-tier (MUL-4910) Follow-up to the immediate-retry fix: make the connection-closed retry a three-tier schedule — first run + immediate retry + one retry deferred ~5s — so a blip that survives the immediate retry gets a short cooldown before the final attempt instead of firing back-to-back. Reuses the existing deferred/fire_at primitive (the comment-routing escalation mechanism) rather than adding new infrastructure: CreateRetryTask gains an optional fire_at; when set, the child is inserted 'deferred' and the existing PromoteDueDeferredTasksForRuntime sweeper — already run promote-first on every claim poll — flips it to 'queued' at fire_at. No migration, no claim-query change, no daemon change. - retryAttemptCeiling: raise provider_network's ceiling to 3 (other reasons keep max_attempts=2); applied in both retryEligible and MaybeRetryFailedTask's budget pre-check so the primary and sweeper paths agree. - retryDelayForAttempt: only provider_network's final attempt is deferred (5s); every other retry — including provider_network's first — stays immediate. - FailTask + MaybeRetryFailedTask pass fire_at and skip the queued broadcast/notify for a deferred child (promotion emits them at fire time). Timing: the deferred child fires on the first claim poll at/after fire_at, so >= 5s; on an otherwise-idle runtime it can stretch to the poll interval — the same behaviour deferred escalations already have. Tests: pure schedule/eligibility coverage (TestProviderNetworkRetrySchedule) plus a DB test asserting fire_at controls deferred-vs-queued, attempt=3, and resume-safety. Co-authored-by: multica-agent <github@multica.ai> * fix(task): persist reason-aware retry budget; respect max_attempts=1 disable (MUL-4910) Addresses the pre-merge review must-fix: retryAttemptCeiling raised provider_network to 3 unconditionally, which (1) overrode the max_attempts=1 "auto-retry disabled" contract and (2) persisted a self-contradictory child (attempt=3, max_attempts=2) that leaks to the task API, so a naive attempt < max_attempts consumer would misjudge the budget. - retryAttemptCeiling now returns taskMaxAttempts unchanged when it is <= 1 (disabled stays disabled) and only ever WIDENS otherwise (max(col, 3) for provider_network) — a higher configured budget is kept. - CreateRetryTask takes an optional max_attempts; FailTask and MaybeRetryFailedTask write the reason-aware effective ceiling into the child so the whole retry chain self-describes (attempt=3, max_attempts=3). NULL inherits the parent column, so non-provider_network reasons are unchanged. Tests: - pure: ceiling widens to 3, keeps a higher budget, and never revives a disabled (max_attempts=1) task; eligibility rejects the disabled case. - DB (end-to-end FailTask): default budget → deferred final child at attempt=3/max_attempts=3; first failure → immediate child; max_attempts=1 → no child. Plus CreateRetryTask persists the passed budget / inherits on NULL. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Bohan-J <bohan@devv.ai> Co-authored-by: multica-agent <github@multica.ai>	2026-07-17 16:47:59 +08:00
Rusty Raven	7356b56a10	fix(taskfailure): anchor HTTP status-code matches to digit boundaries (#5275) ... Anchor the 401/402/403/429/529 HTTP status-code matches to digit boundaries so embedded numbers (e.g. "402913 tokens", "15290ms") no longer misclassify process/unknown failures as provider errors and skew failure observability. Mirrors the existing 5xx anchoring (providerHTTP5xxRe). Fixes #5271 MUL-4422	2026-07-12 13:41:53 +08:00
Multica Eve	10afd1af1b	feat(server): introduce pkg/taskfailure classifier and switch in-flight failure_reason writes (MUL-2946) (#3693) ... Lift MUL-1949's offline backfill failure_reason taxonomy into a shared in-flight classifier so the agent_task_queue.failure_reason column is written with refined values (provider_auth_or_access, context_overflow, provider_capacity_or_rate_limit, …) at write time rather than waiting on SQL backfill to re-classify after the fact. PR1 of the Grafana board plan in MUL-2328 — the upcoming PR2 reuses pkg/taskfailure.AllReasons() to pre-warm the Prometheus failure_reason label set. * server/pkg/taskfailure: new package with the canonical 21 Reason constants (7 platform-side + 14 agent_error.* sub-reasons), AllReasons() returning a defensive copy, IsAgentError() prefix check, and Classify(rawError) Reason mirroring the SQL CASE rules from MUL-1949 (db-boy's analysis). 100% statement coverage. * server/internal/daemon/daemon.go: route the 'agent_error' coarse fallback paths (StartTask error, runTask early-return error, CompleteTask permanent rejection, reportTaskResult default branch) and the executeAndDrain default error case (chained after classifyPoisonedError) through taskfailure.Classify so blocked / timeout / unknown-status results all carry a refined reason on the wire. * server/internal/service/task.go: FailTask classifies errMsg when the daemon-supplied failureReason is empty, eliminating the legacy COALESCE(.., 'agent_error') landing. * server/internal/daemon/poisoned.go: alias FailureReasonIterationLimit and FailureReasonAPIInvalidRequest to the canonical taskfailure constants. agent_fallback_message and codex_semantic_inactivity are pre-existing operational reasons not in the canonical 21 — kept as literals for now and revisited in a follow-up PR. Backfill SQL from MUL-1949 stays as the authoritative offline source of truth; this PR keeps the in-flight classifier in lock-step with the SQL CASE expression so historical and future rows share the same taxonomy. No behavior change for the platform-side reasons (queued_expired, runtime_offline, runtime_recovery, timeout, etc.) which already align with the canonical set. Co-authored-by: Eve <eve@multica-ai.local> Co-authored-by: multica-agent <github@multica.ai>	2026-06-03 13:52:56 +08:00