mirror of
https://github.com/multica-ai/multica.git
synced 2026-08-10 14:58:25 +02:00
* feat(usage): add error/failure visibility to the Usage dashboard The Usage page could only answer "how much did we spend"; nothing on it showed how often agents fail, what kind of failure it was, or which agent is responsible. Operators had to open failed tasks one at a time to spot a pattern. `agent_task_queue.failure_reason` already carries the refined 21-value taxonomy from server/pkg/taskfailure, so this is a read path over data that already exists. Backend — two rollups, both scoped by workspace/project/window like the existing dashboard endpoints: GET /api/dashboard/failures/daily per-(date, failure_reason) GET /api/dashboard/failures/by-agent per-(agent, failure_reason) They return every terminal task, not just failures: the `failure_reason: ""` row carries the succeeded count. That is what makes the error rate's denominator share filters with its numerator. The run-time rollups can't serve as that denominator — they require `started_at IS NOT NULL`, so a task that expired in the queue (the signature of a runtime outage) contributes nothing to their failed_count. A failed row with an empty reason column lands in an `unclassified` bucket rather than being mistaken for a success. Frontend: - "Errors" joins the trend toggle, daily and weekly, stacked by failure class with the bucket's error rate in the tooltip. - An Errors card breaks the window down by class and by agent, with the raw failure_reason strings behind a disclosure (unlocalised — an operator pastes them into a log search). Each agent row links to its Work tab, which lists the actual failed runs. - The 21 backend reasons fold into 7 display classes in @multica/core/dashboard. Unknown reasons — including ones from a backend newer than the client — land in "other" instead of being dropped, so the class totals always reconcile with the failure count. The Tasks KPI tile is deliberately left alone: its value counts started tasks only, so quoting the failure rollup's larger count there would put two denominators in one tile. The Errors card states its rate with the denominator spelled out instead. Migration 225 adds a partial index on agent_task_queue(completed_at) for terminal statuses. The table had no completed_at index at all, so the two pre-existing run-time rollups were already scanning it; these two new queries would have doubled that. Closes #4429 (MUL-5352) Co-authored-by: multica-agent <github@multica.ai> * fix(usage): correct the Errors drill-down, window and agent exposure Review findings on PR #5991. 1. The drill-down pointed at the wrong page. `?view=work` renders ActorIssuesPanel — the issues assigned to the agent — while its runs live in the Overview pane's ActivityTab. Link to Overview. That page also could not show why a run failed: `failureReasonLabel` was a `Record<TaskFailureReason, string>` indexed with a cast to the old 6-value coarse enum, so every refined reason the backend has written since MUL-1949 resolved to `undefined`. It is now a function over the full 21-value taxonomy plus the legacy coarse values, falling back to the raw wire string for anything newer than the client. Fixes the issue execution log too, which had the same cast. 2. The Errors card covered one more calendar day than the chart above it. `parseSinceParamInTZ` returns N+1 days of headroom on purpose and the dashboard trims the surplus client-side — but only a series carrying a date can be trimmed that way. Totals / classes / reasons now derive from the date-bucketed rollup after that trim, and the per-agent rollup (which has no date to trim on) closes its window server-side via a new `parseExactSinceParamInTZ`. At days=1 the card previously reported yesterday's failures beside a chart showing none. 3. The top-offenders list leaked agents the viewer cannot see. The failure rollups are workspace-scoped and deliberately skip per-agent visibility, but the agent list they are joined against does not — members only see a private agent when they own it or are owner/admin. `name ?? row.agentId` therefore rendered a bare UUID along with that agent's failure count, rate and dominant error class. Unresolvable agents now fold into one anonymous row, and the renderer never falls back to an id. Stricter than `bucketUnknownAgentRows` while the agent list loads: a transient flash of UUIDs is the leak, not a cosmetic glitch. Also from the review: the Errors tooltip echoed the raw Recharts dataKey ("rate_limit") instead of the translated label the legend already carries. Not changed — the schema's `failure_reason` default stays `""`. Defaulting a missing field to a failure bucket guards against a deflated rate, but the realistic drift is `omitempty` on the Go struct tag, which would strip the field from exactly the SUCCESS rows and read as a 100% error rate. Added TestDashboardFailureWireContractKeepsEmptyReason to pin that the server always emits the field, which is the assumption the default rests on. Co-authored-by: multica-agent <github@multica.ai> * fix(usage): renumber migration and fix the anonymous bucket's failure class Review findings on PR #5991, round 2. 1. Migration prefix 225 collided with `225_chat_message_channel_media_pending`, which landed on main while this branch was open — backend CI failed on TestMigrationNumericPrefixesStayUniqueAfterLegacySet. Merged main and renumbered to 231; main now carries 225 through 230, so 226 is taken too. 2. The anonymous "Other agents" bucket could announce the wrong failure class. It merged rows that had ALREADY collapsed to one dominant class per agent, then credited each agent's entire failure count to that class. An agent failing auth 6 / timeout 5 contributed 11 to auth and 0 to timeout, so a bucket whose real composition was timeout 15 / auth 6 rendered as Auth. Fixed by anonymizing the raw per-(agent, reason) rows instead: the sentinel becomes just another agent_id and `aggregateAgentFailures` computes its classes from real counts. That also deletes the parallel bucketing pass — one identity rewrite replaces it. `knownAgentIds` moves up to where both consumers can see it. Also from the review: - The wire-contract test decoded both payloads into one map. json.Unmarshal merges into a non-nil map rather than resetting it, so a residual failure_reason from the first case could have masked an omitempty regression in the second — exactly what the test is meant to catch. Now table-driven with a fresh map per case. - A test comment still described the drill-down as pointing at the Work tab. Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Bohan-J <bohan@devv.ai> Co-authored-by: multica-agent <github@multica.ai>
613 lines
21 KiB
TypeScript
613 lines
21 KiB
TypeScript
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||
import {
|
||
aggregateAgentFailures,
|
||
anonymizeUnresolvedAgentRows,
|
||
UNRESOLVED_AGENTS_ROW_ID,
|
||
aggregateAgentTokens,
|
||
aggregateDailyCost,
|
||
aggregateDailyErrors,
|
||
aggregateFailureClasses,
|
||
aggregateFailureReasons,
|
||
aggregateWeeklyErrors,
|
||
aggregateWeeklyTasks,
|
||
aggregateWeeklyTime,
|
||
bucketUnknownAgentRows,
|
||
computeDailyTotals,
|
||
computeFailureTotals,
|
||
DELETED_AGENTS_ROW_ID,
|
||
formatDuration,
|
||
mergeAgentDashboardRows,
|
||
} from "./utils";
|
||
|
||
describe("aggregateDailyCost", () => {
|
||
it("collapses multiple rows per day into one stack and sorts by date asc", () => {
|
||
const result = aggregateDailyCost([
|
||
{
|
||
date: "2026-05-10",
|
||
provider: "claude",
|
||
model: "claude-sonnet-4-6",
|
||
input_tokens: 1_000_000,
|
||
output_tokens: 500_000,
|
||
cache_read_tokens: 0,
|
||
cache_write_tokens: 0,
|
||
task_count: 3,
|
||
},
|
||
{
|
||
date: "2026-05-09",
|
||
provider: "claude",
|
||
model: "claude-sonnet-4-6",
|
||
input_tokens: 1_000_000,
|
||
output_tokens: 0,
|
||
cache_read_tokens: 0,
|
||
cache_write_tokens: 0,
|
||
task_count: 1,
|
||
},
|
||
]);
|
||
|
||
// Sort: oldest day first.
|
||
expect(result.map((r) => r.date)).toEqual(["2026-05-09", "2026-05-10"]);
|
||
// claude-sonnet-4-6: input $3/M, output $15/M.
|
||
// 2026-05-09 → 1M input × $3 = $3 input, $0 output, $0 cache.
|
||
expect(result[0]).toMatchObject({ input: 3, output: 0, cacheWrite: 0, total: 3 });
|
||
// 2026-05-10 → $3 input + (0.5M × $15) = $7.5 output. Total $10.5.
|
||
expect(result[1]).toMatchObject({ input: 3, output: 7.5, cacheWrite: 0, total: 10.5 });
|
||
});
|
||
|
||
it("treats unmapped models as zero-cost", () => {
|
||
const result = aggregateDailyCost([
|
||
{
|
||
date: "2026-05-10",
|
||
provider: "claude",
|
||
model: "made-up-model",
|
||
input_tokens: 999_999_999,
|
||
output_tokens: 0,
|
||
cache_read_tokens: 0,
|
||
cache_write_tokens: 0,
|
||
task_count: 0,
|
||
},
|
||
]);
|
||
expect(result[0]?.total).toBe(0);
|
||
});
|
||
});
|
||
|
||
describe("aggregateAgentTokens", () => {
|
||
it("folds per-(agent, model) rows into per-agent totals and sorts by cost desc", () => {
|
||
const rows = aggregateAgentTokens([
|
||
{
|
||
agent_id: "small-spender",
|
||
provider: "claude",
|
||
model: "claude-sonnet-4-6",
|
||
input_tokens: 100_000,
|
||
output_tokens: 0,
|
||
cache_read_tokens: 0,
|
||
cache_write_tokens: 0,
|
||
task_count: 1,
|
||
},
|
||
{
|
||
agent_id: "big-spender",
|
||
provider: "claude",
|
||
model: "claude-sonnet-4-6",
|
||
input_tokens: 5_000_000,
|
||
output_tokens: 0,
|
||
cache_read_tokens: 0,
|
||
cache_write_tokens: 0,
|
||
task_count: 3,
|
||
},
|
||
{
|
||
agent_id: "big-spender",
|
||
provider: "claude",
|
||
model: "claude-haiku-4-5",
|
||
input_tokens: 1_000_000,
|
||
output_tokens: 0,
|
||
cache_read_tokens: 0,
|
||
cache_write_tokens: 0,
|
||
task_count: 2,
|
||
},
|
||
]);
|
||
|
||
expect(rows.map((r) => r.agentId)).toEqual(["big-spender", "small-spender"]);
|
||
expect(rows[0]?.taskCount).toBe(5);
|
||
// big-spender across two models — verify cost > small-spender's.
|
||
expect(rows[0]!.cost).toBeGreaterThan(rows[1]!.cost);
|
||
});
|
||
});
|
||
|
||
describe("computeDailyTotals", () => {
|
||
it("sums tokens across rows and adds estimated cost", () => {
|
||
const totals = computeDailyTotals([
|
||
{
|
||
date: "2026-05-10",
|
||
provider: "claude",
|
||
model: "claude-sonnet-4-6",
|
||
input_tokens: 1_000_000,
|
||
output_tokens: 0,
|
||
cache_read_tokens: 0,
|
||
cache_write_tokens: 0,
|
||
task_count: 2,
|
||
},
|
||
{
|
||
date: "2026-05-09",
|
||
provider: "claude",
|
||
model: "claude-sonnet-4-6",
|
||
input_tokens: 2_000_000,
|
||
output_tokens: 0,
|
||
cache_read_tokens: 0,
|
||
cache_write_tokens: 0,
|
||
task_count: 3,
|
||
},
|
||
]);
|
||
expect(totals.input).toBe(3_000_000);
|
||
expect(totals.cost).toBe(9); // 3M × $3/M
|
||
expect(totals.taskCount).toBe(5);
|
||
});
|
||
});
|
||
|
||
describe("mergeAgentDashboardRows", () => {
|
||
it("uses run-time rollup's per-agent task count, not the token sum", () => {
|
||
// Token rollup returns two (agent, model) rows for the same task
|
||
// (the agent ran one task that touched two models). The token-side
|
||
// aggregator sums per-row task_count and lands at 2; the run-time
|
||
// rollup correctly reports the underlying distinct count of 1.
|
||
const tokenRows = [
|
||
{
|
||
agentId: "agent-a",
|
||
tokens: 3_000_000,
|
||
cost: 12,
|
||
taskCount: 2, // overcounted because (model-1: 1) + (model-2: 1)
|
||
},
|
||
];
|
||
const runTimeRows = [
|
||
{
|
||
agent_id: "agent-a",
|
||
total_seconds: 600,
|
||
task_count: 1, // truth: one task touched both models
|
||
failed_count: 0,
|
||
},
|
||
];
|
||
const merged = mergeAgentDashboardRows(tokenRows, runTimeRows);
|
||
expect(merged).toHaveLength(1);
|
||
expect(merged[0]!.taskCount).toBe(1);
|
||
expect(merged[0]!.seconds).toBe(600);
|
||
});
|
||
|
||
it("falls back to token count when no run-time row exists (in-flight task)", () => {
|
||
// Tokens reported mid-run; task hasn't terminated yet so the run-time
|
||
// rollup is silent on this agent. Keep the token-side estimate
|
||
// instead of dropping the agent from the table entirely.
|
||
const merged = mergeAgentDashboardRows(
|
||
[{ agentId: "agent-b", tokens: 100, cost: 0.5, taskCount: 1 }],
|
||
[],
|
||
);
|
||
expect(merged[0]!.taskCount).toBe(1);
|
||
expect(merged[0]!.seconds).toBe(0);
|
||
});
|
||
|
||
it("includes agents that have run-time but no tokens", () => {
|
||
// Task errored before reporting any usage — run-time row exists but
|
||
// there's no corresponding token row. Agent must still appear on the
|
||
// list with zeroed-out token columns.
|
||
const merged = mergeAgentDashboardRows(
|
||
[],
|
||
[{ agent_id: "agent-c", total_seconds: 30, task_count: 1, failed_count: 1 }],
|
||
);
|
||
expect(merged).toHaveLength(1);
|
||
expect(merged[0]!.tokens).toBe(0);
|
||
expect(merged[0]!.cost).toBe(0);
|
||
expect(merged[0]!.taskCount).toBe(1);
|
||
});
|
||
|
||
it("sorts by cost desc with run-time as a tiebreaker", () => {
|
||
const merged = mergeAgentDashboardRows(
|
||
[
|
||
{ agentId: "low", tokens: 100, cost: 1, taskCount: 1 },
|
||
{ agentId: "high", tokens: 100, cost: 9, taskCount: 1 },
|
||
{ agentId: "zero-cost-long", tokens: 0, cost: 0, taskCount: 0 },
|
||
],
|
||
[
|
||
{ agent_id: "zero-cost-long", total_seconds: 1000, task_count: 5, failed_count: 0 },
|
||
],
|
||
);
|
||
expect(merged.map((r) => r.agentId)).toEqual(["high", "low", "zero-cost-long"]);
|
||
});
|
||
});
|
||
|
||
describe("bucketUnknownAgentRows", () => {
|
||
const live = { agentId: "live", tokens: 100, cost: 1, seconds: 10, taskCount: 1 };
|
||
const archived = {
|
||
agentId: "archived",
|
||
tokens: 80,
|
||
cost: 0.8,
|
||
seconds: 8,
|
||
taskCount: 2,
|
||
};
|
||
const deletedA = {
|
||
agentId: "deleted-a",
|
||
tokens: 50,
|
||
cost: 0.5,
|
||
seconds: 5,
|
||
taskCount: 1,
|
||
};
|
||
const deletedB = {
|
||
agentId: "deleted-b",
|
||
tokens: 30,
|
||
cost: 0.25,
|
||
seconds: 3,
|
||
taskCount: 4,
|
||
};
|
||
|
||
it("folds every hard-deleted agent into one aggregated bucket row", () => {
|
||
// "deleted-a" / "deleted-b" are absent from the known set — they'd otherwise
|
||
// render as bare UUIDs. They collapse into a single sentinel row.
|
||
const out = bucketUnknownAgentRows(
|
||
[live, deletedA, deletedB],
|
||
new Set(["live"]),
|
||
);
|
||
expect(out.map((r) => r.agentId)).toEqual(["live", DELETED_AGENTS_ROW_ID]);
|
||
const bucket = out.find((r) => r.agentId === DELETED_AGENTS_ROW_ID)!;
|
||
expect(bucket.tokens).toBe(80);
|
||
expect(bucket.cost).toBeCloseTo(0.75);
|
||
// Time/Tasks never attach to the bucket — the run-time rollup inner-joins
|
||
// `agent`, so deleted agents contribute nothing to those columns.
|
||
expect(bucket.seconds).toBe(0);
|
||
expect(bucket.taskCount).toBe(0);
|
||
});
|
||
|
||
it("keeps the bucket total reconciled with the top-line spend", () => {
|
||
// The KPI total counts deleted-agent spend; sum(visible rows) must match it
|
||
// so the breakdown reconciles (MUL-3776).
|
||
const out = bucketUnknownAgentRows(
|
||
[live, deletedA, deletedB],
|
||
new Set(["live"]),
|
||
);
|
||
const visibleCost = out.reduce((s, r) => s + r.cost, 0);
|
||
const kpiCost = [live, deletedA, deletedB].reduce((s, r) => s + r.cost, 0);
|
||
expect(visibleCost).toBeCloseTo(kpiCost);
|
||
});
|
||
|
||
it("keeps archived agents as themselves, never in the bucket", () => {
|
||
// The agent list is fetched with archived included, so archived agents are
|
||
// in the known set and stay on the board under their own id.
|
||
const out = bucketUnknownAgentRows(
|
||
[live, archived, deletedA],
|
||
new Set(["live", "archived"]),
|
||
);
|
||
expect(out.map((r) => r.agentId)).toEqual([
|
||
"live",
|
||
"archived",
|
||
DELETED_AGENTS_ROW_ID,
|
||
]);
|
||
});
|
||
|
||
it("adds no bucket row when every agent is known", () => {
|
||
const out = bucketUnknownAgentRows([live, archived], new Set(["live", "archived"]));
|
||
expect(out.map((r) => r.agentId)).toEqual(["live", "archived"]);
|
||
});
|
||
|
||
it("keeps every row untouched while the agent list is still loading (null set)", () => {
|
||
const out = bucketUnknownAgentRows([live, deletedA], null);
|
||
expect(out.map((r) => r.agentId)).toEqual(["live", "deleted-a"]);
|
||
});
|
||
});
|
||
|
||
describe("formatDuration", () => {
|
||
it("formats seconds-only durations", () => {
|
||
expect(formatDuration(45, "<1m")).toBe("45s");
|
||
});
|
||
it("formats minutes and seconds when under one hour", () => {
|
||
expect(formatDuration(150, "<1m")).toBe("2m 30s");
|
||
expect(formatDuration(60, "<1m")).toBe("1m");
|
||
});
|
||
it("formats hours and minutes when under one day", () => {
|
||
expect(formatDuration(3 * 3600 + 17 * 60, "<1m")).toBe("3h 17m");
|
||
expect(formatDuration(3600, "<1m")).toBe("1h");
|
||
});
|
||
it("formats days and hours when more than 24 hours", () => {
|
||
expect(formatDuration(2 * 86400 + 5 * 3600, "<1m")).toBe("2d 5h");
|
||
});
|
||
it("falls back to the supplied label for sub-second durations", () => {
|
||
expect(formatDuration(0, "<1m")).toBe("<1m");
|
||
expect(formatDuration(0.4, "<1m")).toBe("<1m");
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// Weekly run-time / tasks aggregation. Mirrors the runtimes-side
|
||
// aggregateByWeek tests: trailing N calendar weeks anchored at today-in-tz,
|
||
// pre-zeroed buckets, partial-week metadata, and rows outside the window
|
||
// dropped. We assert the same invariants on the workspace dashboard helpers
|
||
// so all four metrics behave consistently when the user toggles Weekly.
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe("aggregateWeeklyTime", () => {
|
||
beforeEach(() => {
|
||
vi.useFakeTimers();
|
||
});
|
||
afterEach(() => {
|
||
vi.useRealTimers();
|
||
});
|
||
|
||
it("folds per-day run-time rows into Mon-anchored weekly totals", () => {
|
||
// 2026-05-19 is a Tuesday → current week is Mon=05-18..Sun=05-24.
|
||
vi.setSystemTime(new Date("2026-05-19T12:00:00Z"));
|
||
const rows = [
|
||
{ date: "2026-05-11", total_seconds: 100, task_count: 0, failed_count: 0 },
|
||
{ date: "2026-05-17", total_seconds: 50, task_count: 0, failed_count: 0 },
|
||
{ date: "2026-05-18", total_seconds: 25, task_count: 0, failed_count: 0 },
|
||
];
|
||
const result = aggregateWeeklyTime(rows, "UTC", 2);
|
||
expect(result).toHaveLength(2);
|
||
expect(result[0]).toMatchObject({
|
||
weekStart: "2026-05-11",
|
||
weekEnd: "2026-05-17",
|
||
totalSeconds: 150,
|
||
partial: false,
|
||
daysCovered: 7,
|
||
});
|
||
expect(result[1]).toMatchObject({
|
||
weekStart: "2026-05-18",
|
||
totalSeconds: 25,
|
||
partial: true,
|
||
daysCovered: 2, // Mon + Tue
|
||
});
|
||
});
|
||
|
||
it("drops rows that fall outside the trailing window and keeps empty buckets", () => {
|
||
// Same MUL-2382 sparse-data regression we caught on the runtimes side:
|
||
// an old populated week must not surface when the requested window
|
||
// doesn't include it; in-range empty weeks must remain as zero buckets.
|
||
vi.setSystemTime(new Date("2026-05-19T12:00:00Z"));
|
||
const rows = [
|
||
// 2026-04-13 is a Monday — exactly one week earlier than the oldest
|
||
// in-range week (Mon=04-20) for a 5-week trailing window.
|
||
{ date: "2026-04-13", total_seconds: 999, task_count: 0, failed_count: 0 },
|
||
];
|
||
const result = aggregateWeeklyTime(rows, "UTC", 5);
|
||
expect(result.map((w) => w.weekStart)).toEqual([
|
||
"2026-04-20",
|
||
"2026-04-27",
|
||
"2026-05-04",
|
||
"2026-05-11",
|
||
"2026-05-18",
|
||
]);
|
||
for (const w of result) expect(w.totalSeconds).toBe(0);
|
||
});
|
||
});
|
||
|
||
describe("aggregateWeeklyTasks", () => {
|
||
beforeEach(() => {
|
||
vi.useFakeTimers();
|
||
});
|
||
afterEach(() => {
|
||
vi.useRealTimers();
|
||
});
|
||
|
||
it("splits completed and failed counts per calendar week", () => {
|
||
vi.setSystemTime(new Date("2026-05-19T12:00:00Z"));
|
||
const rows = [
|
||
{ date: "2026-05-12", total_seconds: 0, task_count: 5, failed_count: 1 },
|
||
{ date: "2026-05-18", total_seconds: 0, task_count: 3, failed_count: 0 },
|
||
];
|
||
const result = aggregateWeeklyTasks(rows, "UTC", 2);
|
||
expect(result[0]).toMatchObject({
|
||
weekStart: "2026-05-11",
|
||
completed: 4,
|
||
failed: 1,
|
||
});
|
||
expect(result[1]).toMatchObject({
|
||
weekStart: "2026-05-18",
|
||
completed: 3,
|
||
failed: 0,
|
||
partial: true,
|
||
});
|
||
});
|
||
});
|
||
|
||
// ---------------------------------------------------------------------------
|
||
// Failure aggregations
|
||
//
|
||
// The rollups ship succeeded rows too, marked by `failure_reason: ""`. Every
|
||
// test below leans on that: the succeeded row is what makes an error *rate*
|
||
// possible, and mishandling it is the failure mode with the worst blast
|
||
// radius — a rate that reads 100% when nothing is wrong.
|
||
// ---------------------------------------------------------------------------
|
||
|
||
describe("aggregateDailyErrors", () => {
|
||
it("stacks failures by class and keeps the succeeded rows as the denominator", () => {
|
||
const result = aggregateDailyErrors([
|
||
{ date: "2026-05-10", failure_reason: "", task_count: 8 },
|
||
{
|
||
date: "2026-05-10",
|
||
failure_reason: "agent_error.provider_auth_or_access",
|
||
task_count: 2,
|
||
},
|
||
{ date: "2026-05-10", failure_reason: "timeout", task_count: 1 },
|
||
{ date: "2026-05-09", failure_reason: "", task_count: 4 },
|
||
]);
|
||
|
||
expect(result.map((r) => r.date)).toEqual(["2026-05-09", "2026-05-10"]);
|
||
expect(result[1]).toMatchObject({
|
||
auth: 2,
|
||
timeout: 1,
|
||
rate_limit: 0,
|
||
failed: 3,
|
||
total: 11,
|
||
});
|
||
// A day with only successes still renders a bar slot, at zero height.
|
||
expect(result[0]).toMatchObject({ failed: 0, total: 4 });
|
||
});
|
||
|
||
it("folds a reason this build has never seen into 'other' rather than dropping it", () => {
|
||
const [row] = aggregateDailyErrors([
|
||
{ date: "2026-05-10", failure_reason: "agent_error.from_the_future", task_count: 5 },
|
||
]);
|
||
expect(row).toMatchObject({ other: 5, failed: 5, total: 5 });
|
||
});
|
||
});
|
||
|
||
describe("aggregateWeeklyErrors", () => {
|
||
beforeEach(() => {
|
||
vi.useFakeTimers();
|
||
});
|
||
afterEach(() => {
|
||
vi.useRealTimers();
|
||
});
|
||
|
||
it("buckets per calendar week and pre-zeroes weeks with no terminal tasks", () => {
|
||
vi.setSystemTime(new Date("2026-05-19T12:00:00Z"));
|
||
const result = aggregateWeeklyErrors(
|
||
[
|
||
{ date: "2026-05-12", failure_reason: "runtime_offline", task_count: 2 },
|
||
{ date: "2026-05-12", failure_reason: "", task_count: 6 },
|
||
],
|
||
"UTC",
|
||
2,
|
||
);
|
||
|
||
expect(result[0]).toMatchObject({
|
||
weekStart: "2026-05-11",
|
||
runtime: 2,
|
||
failed: 2,
|
||
total: 8,
|
||
});
|
||
expect(result[1]).toMatchObject({
|
||
weekStart: "2026-05-18",
|
||
failed: 0,
|
||
total: 0,
|
||
partial: true,
|
||
});
|
||
});
|
||
});
|
||
|
||
describe("computeFailureTotals", () => {
|
||
it("excludes the succeeded bucket from the numerator but not the denominator", () => {
|
||
expect(
|
||
computeFailureTotals([
|
||
{ failure_reason: "", task_count: 9 },
|
||
{ failure_reason: "timeout", task_count: 1 },
|
||
]),
|
||
).toEqual({ failed: 1, total: 10, rate: 0.1 });
|
||
});
|
||
|
||
it("reports rate 0 rather than dividing by zero on an empty window", () => {
|
||
expect(computeFailureTotals([])).toEqual({ failed: 0, total: 0, rate: 0 });
|
||
});
|
||
});
|
||
|
||
describe("aggregateFailureClasses / aggregateFailureReasons", () => {
|
||
const rows = [
|
||
{ failure_reason: "", task_count: 20 },
|
||
{ failure_reason: "agent_error.provider_quota_limit", task_count: 3 },
|
||
{ failure_reason: "agent_error.provider_capacity_or_rate_limit", task_count: 4 },
|
||
{ failure_reason: "timeout", task_count: 5 },
|
||
];
|
||
|
||
it("merges reasons that share a class and ranks by count desc", () => {
|
||
expect(aggregateFailureClasses(rows)).toEqual([
|
||
{ failureClass: "rate_limit", count: 7 },
|
||
{ failureClass: "timeout", count: 5 },
|
||
]);
|
||
});
|
||
|
||
it("keeps raw reasons separate so an operator can search the exact string", () => {
|
||
expect(aggregateFailureReasons(rows)).toEqual([
|
||
{ reason: "timeout", failureClass: "timeout", count: 5 },
|
||
{
|
||
reason: "agent_error.provider_capacity_or_rate_limit",
|
||
failureClass: "rate_limit",
|
||
count: 4,
|
||
},
|
||
{
|
||
reason: "agent_error.provider_quota_limit",
|
||
failureClass: "rate_limit",
|
||
count: 3,
|
||
},
|
||
]);
|
||
});
|
||
});
|
||
|
||
describe("aggregateAgentFailures", () => {
|
||
it("ranks by failure count, carries the rate, and names the dominant class", () => {
|
||
const result = aggregateAgentFailures([
|
||
{ agent_id: "a", failure_reason: "", task_count: 90 },
|
||
{ agent_id: "a", failure_reason: "timeout", task_count: 10 },
|
||
{ agent_id: "b", failure_reason: "", task_count: 1 },
|
||
{ agent_id: "b", failure_reason: "runtime_offline", task_count: 3 },
|
||
{ agent_id: "b", failure_reason: "timeout", task_count: 1 },
|
||
]);
|
||
|
||
// `a` fails 10% of the time, `b` fails 80% — but `a` is the bigger
|
||
// absolute problem, so it ranks first and the rate rides along.
|
||
expect(result).toEqual([
|
||
{ agentId: "a", failed: 10, total: 100, rate: 0.1, topClass: "timeout" },
|
||
{ agentId: "b", failed: 4, total: 5, rate: 0.8, topClass: "runtime" },
|
||
]);
|
||
});
|
||
|
||
it("drops agents with no failures — the list is triage, not a census", () => {
|
||
expect(
|
||
aggregateAgentFailures([{ agent_id: "clean", failure_reason: "", task_count: 42 }]),
|
||
).toEqual([]);
|
||
});
|
||
});
|
||
|
||
describe("anonymizeUnresolvedAgentRows", () => {
|
||
// Raw per-(agent, reason) rows, which is the shape this operates on. Two
|
||
// agents the viewer cannot resolve, with deliberately conflicting dominant
|
||
// classes — see the counterexample test below.
|
||
const rows = [
|
||
{ agent_id: "visible", failure_reason: "", task_count: 5 },
|
||
{ agent_id: "visible", failure_reason: "timeout", task_count: 5 },
|
||
{
|
||
agent_id: "private-a",
|
||
failure_reason: "agent_error.provider_auth_or_access",
|
||
task_count: 6,
|
||
},
|
||
{ agent_id: "private-a", failure_reason: "timeout", task_count: 5 },
|
||
{ agent_id: "private-b", failure_reason: "timeout", task_count: 10 },
|
||
];
|
||
|
||
it("rewrites unresolvable ids to the sentinel and leaves resolvable ones alone", () => {
|
||
const result = anonymizeUnresolvedAgentRows(rows, new Set(["visible"]));
|
||
|
||
expect(result.map((r) => r.agent_id)).toEqual([
|
||
"visible",
|
||
"visible",
|
||
UNRESOLVED_AGENTS_ROW_ID,
|
||
UNRESOLVED_AGENTS_ROW_ID,
|
||
UNRESOLVED_AGENTS_ROW_ID,
|
||
]);
|
||
// Counts are untouched — only identity is erased.
|
||
expect(result.map((r) => r.task_count)).toEqual([5, 5, 6, 5, 10]);
|
||
});
|
||
|
||
it("keeps the bucket's dominant class honest across merged agents", () => {
|
||
// This is why the rewrite happens on RAW rows. private-a is auth-dominant
|
||
// (6 vs 5) and private-b is timeout-only (10). Merging AFTER aggregation
|
||
// would see only each agent's top class and its total failure count —
|
||
// auth 11, timeout 10 — and label the bucket Auth. The true composition
|
||
// is timeout 15 / auth 6, so it must read Timeout.
|
||
const bucket = aggregateAgentFailures(
|
||
anonymizeUnresolvedAgentRows(rows, new Set(["visible"])),
|
||
).find((r) => r.agentId === UNRESOLVED_AGENTS_ROW_ID);
|
||
|
||
expect(bucket).toMatchObject({ failed: 21, total: 21, topClass: "timeout" });
|
||
});
|
||
|
||
it("anonymizes everything while the agent list is still loading", () => {
|
||
// Deliberately stricter than bucketUnknownAgentRows, which passes rows
|
||
// through on null: a transient flash of raw UUIDs is precisely the leak
|
||
// this function exists to prevent.
|
||
const result = anonymizeUnresolvedAgentRows(rows, null);
|
||
|
||
expect(new Set(result.map((r) => r.agent_id))).toEqual(
|
||
new Set([UNRESOLVED_AGENTS_ROW_ID]),
|
||
);
|
||
});
|
||
|
||
it("returns the input untouched when every agent resolves", () => {
|
||
const known = new Set(["visible", "private-a", "private-b"]);
|
||
// Same reference, not just equal — nothing needed rewriting.
|
||
expect(anonymizeUnresolvedAgentRows(rows, known)).toBe(rows);
|
||
});
|
||
});
|