mirror of
https://github.com/multica-ai/multica.git
synced 2026-07-27 21:33:41 +02:00
4d0475ce89
* feat(usage): add error/failure visibility to the Usage dashboard The Usage page could only answer "how much did we spend"; nothing on it showed how often agents fail, what kind of failure it was, or which agent is responsible. Operators had to open failed tasks one at a time to spot a pattern. `agent_task_queue.failure_reason` already carries the refined 21-value taxonomy from server/pkg/taskfailure, so this is a read path over data that already exists. Backend — two rollups, both scoped by workspace/project/window like the existing dashboard endpoints: GET /api/dashboard/failures/daily per-(date, failure_reason) GET /api/dashboard/failures/by-agent per-(agent, failure_reason) They return every terminal task, not just failures: the `failure_reason: ""` row carries the succeeded count. That is what makes the error rate's denominator share filters with its numerator. The run-time rollups can't serve as that denominator — they require `started_at IS NOT NULL`, so a task that expired in the queue (the signature of a runtime outage) contributes nothing to their failed_count. A failed row with an empty reason column lands in an `unclassified` bucket rather than being mistaken for a success. Frontend: - "Errors" joins the trend toggle, daily and weekly, stacked by failure class with the bucket's error rate in the tooltip. - An Errors card breaks the window down by class and by agent, with the raw failure_reason strings behind a disclosure (unlocalised — an operator pastes them into a log search). Each agent row links to its Work tab, which lists the actual failed runs. - The 21 backend reasons fold into 7 display classes in @multica/core/dashboard. Unknown reasons — including ones from a backend newer than the client — land in "other" instead of being dropped, so the class totals always reconcile with the failure count. The Tasks KPI tile is deliberately left alone: its value counts started tasks only, so quoting the failure rollup's larger count there would put two denominators in one tile. The Errors card states its rate with the denominator spelled out instead. Migration 225 adds a partial index on agent_task_queue(completed_at) for terminal statuses. The table had no completed_at index at all, so the two pre-existing run-time rollups were already scanning it; these two new queries would have doubled that. Closes #4429 (MUL-5352) Co-authored-by: multica-agent <github@multica.ai> * fix(usage): correct the Errors drill-down, window and agent exposure Review findings on PR #5991. 1. The drill-down pointed at the wrong page. `?view=work` renders ActorIssuesPanel — the issues assigned to the agent — while its runs live in the Overview pane's ActivityTab. Link to Overview. That page also could not show why a run failed: `failureReasonLabel` was a `Record<TaskFailureReason, string>` indexed with a cast to the old 6-value coarse enum, so every refined reason the backend has written since MUL-1949 resolved to `undefined`. It is now a function over the full 21-value taxonomy plus the legacy coarse values, falling back to the raw wire string for anything newer than the client. Fixes the issue execution log too, which had the same cast. 2. The Errors card covered one more calendar day than the chart above it. `parseSinceParamInTZ` returns N+1 days of headroom on purpose and the dashboard trims the surplus client-side — but only a series carrying a date can be trimmed that way. Totals / classes / reasons now derive from the date-bucketed rollup after that trim, and the per-agent rollup (which has no date to trim on) closes its window server-side via a new `parseExactSinceParamInTZ`. At days=1 the card previously reported yesterday's failures beside a chart showing none. 3. The top-offenders list leaked agents the viewer cannot see. The failure rollups are workspace-scoped and deliberately skip per-agent visibility, but the agent list they are joined against does not — members only see a private agent when they own it or are owner/admin. `name ?? row.agentId` therefore rendered a bare UUID along with that agent's failure count, rate and dominant error class. Unresolvable agents now fold into one anonymous row, and the renderer never falls back to an id. Stricter than `bucketUnknownAgentRows` while the agent list loads: a transient flash of UUIDs is the leak, not a cosmetic glitch. Also from the review: the Errors tooltip echoed the raw Recharts dataKey ("rate_limit") instead of the translated label the legend already carries. Not changed — the schema's `failure_reason` default stays `""`. Defaulting a missing field to a failure bucket guards against a deflated rate, but the realistic drift is `omitempty` on the Go struct tag, which would strip the field from exactly the SUCCESS rows and read as a 100% error rate. Added TestDashboardFailureWireContractKeepsEmptyReason to pin that the server always emits the field, which is the assumption the default rests on. Co-authored-by: multica-agent <github@multica.ai> * fix(usage): renumber migration and fix the anonymous bucket's failure class Review findings on PR #5991, round 2. 1. Migration prefix 225 collided with `225_chat_message_channel_media_pending`, which landed on main while this branch was open — backend CI failed on TestMigrationNumericPrefixesStayUniqueAfterLegacySet. Merged main and renumbered to 231; main now carries 225 through 230, so 226 is taken too. 2. The anonymous "Other agents" bucket could announce the wrong failure class. It merged rows that had ALREADY collapsed to one dominant class per agent, then credited each agent's entire failure count to that class. An agent failing auth 6 / timeout 5 contributed 11 to auth and 0 to timeout, so a bucket whose real composition was timeout 15 / auth 6 rendered as Auth. Fixed by anonymizing the raw per-(agent, reason) rows instead: the sentinel becomes just another agent_id and `aggregateAgentFailures` computes its classes from real counts. That also deletes the parallel bucketing pass — one identity rewrite replaces it. `knownAgentIds` moves up to where both consumers can see it. Also from the review: - The wire-contract test decoded both payloads into one map. json.Unmarshal merges into a non-nil map rather than resetting it, so a residual failure_reason from the first case could have masked an omitempty regression in the second — exactly what the test is meant to catch. Now table-driven with a fresh map per case. - A test comment still described the drill-down as pointing at the Work tab. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Bohan-J <bohan@devv.ai> Co-authored-by: multica-agent <github@multica.ai>
66 lines
3.0 KiB
TypeScript
66 lines
3.0 KiB
TypeScript
// Human-readable copy for the back-end task failure reason. Surfaced in the
|
|
// agent detail Recent Work list and the issue execution log — the two places
|
|
// the front-end exposes failure_reason directly to the user, and the
|
|
// destination the Usage page's Errors breakdown drills into.
|
|
//
|
|
// Lives next to the consuming tab (rather than in agents/presence) because
|
|
// failed tasks no longer have a top-level workload state; failure context
|
|
// is purely a detail-page concern now.
|
|
//
|
|
// Covers the canonical taxonomy in server/pkg/taskfailure — 7 platform-side
|
|
// reasons plus 14 `agent_error.*` sub-reasons — and the pre-MUL-1949 coarse
|
|
// values still present on historical rows. This used to be a
|
|
// `Record<TaskFailureReason, string>` indexed with a cast, which silently
|
|
// resolved to `undefined` for every refined reason the backend has written
|
|
// since MUL-1949: a task that failed on a provider 401 rendered no reason
|
|
// at all.
|
|
const REASON_LABEL: Record<string, string> = {
|
|
// Platform / scheduler side.
|
|
queued_expired: "Expired in queue",
|
|
runtime_offline: "Daemon offline",
|
|
runtime_recovery: "Daemon restarted",
|
|
timeout: "Task timed out",
|
|
iteration_limit: "Hit the iteration limit",
|
|
agent_blocked: "Waiting on human input",
|
|
api_invalid_request: "Rejected by the model API",
|
|
|
|
// Agent process side — provider.
|
|
"agent_error.provider_auth_or_access": "Provider auth failed",
|
|
"agent_error.provider_quota_limit": "Provider quota exhausted",
|
|
"agent_error.provider_capacity_or_rate_limit": "Rate limited by provider",
|
|
"agent_error.provider_server_error": "Provider server error",
|
|
"agent_error.provider_network": "Network error reaching provider",
|
|
|
|
// Agent process side — agent / runner.
|
|
"agent_error.process_failure": "Agent process crashed",
|
|
"agent_error.empty_or_unparseable_output": "Agent returned no usable output",
|
|
"agent_error.agent_timeout": "Agent timed out",
|
|
"agent_error.context_overflow": "Context window exceeded",
|
|
"agent_error.missing_config": "Missing API key or configuration",
|
|
"agent_error.model_not_found_or_unavailable": "Model unavailable",
|
|
"agent_error.runtime_version_unsupported": "Runner CLI version unsupported",
|
|
"agent_error.runtime_missing_executable": "Runner CLI not installed",
|
|
"agent_error.unknown": "Agent execution error",
|
|
|
|
// Pre-MUL-1949 coarse values, still present on historical rows.
|
|
agent_error: "Agent execution error",
|
|
codex_semantic_inactivity: "Codex semantic inactivity timeout",
|
|
manual: "Cancelled by user",
|
|
};
|
|
|
|
/**
|
|
* Label for a `failure_reason`, or `null` when there is nothing to show.
|
|
*
|
|
* `failure_reason` is an open string, not a closed enum: the taxonomy grows
|
|
* as classifier rules land, and an installed desktop client will meet
|
|
* reasons its build predates. Those fall back to the raw wire value rather
|
|
* than to nothing — machine-y, but truthful, and it is the exact string an
|
|
* operator would paste into a log search.
|
|
*/
|
|
export function failureReasonLabel(
|
|
reason: string | null | undefined,
|
|
): string | null {
|
|
if (!reason) return null;
|
|
return REASON_LABEL[reason] ?? reason;
|
|
}
|