4 Commits

Author	SHA1	Message	Date
Multica Eve	eb067ff077	fix(server): aggregate task_usage into daily rollup table to cut DB load (#2256) ... * fix(server): aggregate task_usage into daily rollup table to cut DB load ListRuntimeUsage previously did a SUM(...) GROUP BY DATE(created_at), provider, model over the raw task_usage stream once per runtime row on the runtimes list and once per detail page load, scaling O(events) per call. This is the hot read path responsible for sustained load on Postgres. Switch the read path to a materialized daily rollup table maintained by a pg_cron job: - 072_task_usage_daily_rollup: schema for task_usage_daily + task_usage_rollup_state, plus rollup_task_usage_daily_window(p_from, p_to) (window primitive used by both cron and offline backfill, idempotent via ON CONFLICT DO UPDATE adding deltas) and rollup_task_usage_daily() (cron entry point — pg_try_advisory_lock(4242) for serialization, watermark advancement, 5-minute safety lag for late-visible inserts). Also adds idx_task_usage_created_at to help the two lazy endpoints (ListRuntimeUsageByAgent / GetRuntimeUsageByHour) that still hit the raw table. - 073_task_usage_daily_pgcron: CREATE EXTENSION IF NOT EXISTS pg_cron in a DO/EXCEPTION block (mirrors the migration 032 pg_bigm pattern so envs without shared_preload_libraries=pg_cron skip gracefully) and schedules rollup_task_usage_daily() every 5 minutes when the extension is present. - queries/runtime_usage.sql ListRuntimeUsage rewritten to read from task_usage_daily; sqlc regenerated. Other usage queries unchanged. - cmd/backfill_task_usage_daily: one-shot Go command that walks task_usage in monthly slices through rollup_task_usage_daily_window, then stamps the watermark to now()-5m so the cron resumes cleanly. Run once after migrations have applied, before relying on the rollup. - runtime_test.go: TestGetRuntimeUsage_BucketsByUsageTime now invokes rollup_task_usage_daily_window after fixture inserts so the handler sees the rolled-up rows. Synthetic daily rows cleaned up after each test. - runtime_rollup_test.go: new tests covering aggregation correctness, idempotency contract of ON CONFLICT DO UPDATE, and the watermark advancing exactly to now()-5m via the cron entry point. Deployment order: apply migrations → run backfill_task_usage_daily once → pg_cron picks up subsequent windows automatically. Today bucket may be up to ~10 minutes stale (5 min cron + 5 min lag) by design. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * fix(server): make task_usage_daily rollup safe to overlap, replay, and correct Addresses 4 review blockers on the original PR: 1. Cron/backfill double-count race: the rollup function is now idempotent. Window calls find DIRTY KEYS via task_usage.updated_at, then RECOMPUTE each bucket from ground truth and REPLACE the daily row (no more additive ON CONFLICT). Cron and backfill can now overlap safely. 2. Silent pg_cron absence: the read path is gated behind a new USAGE_DAILY_ROLLUP_ENABLED feature flag (default off). The raw task_usage scan is preserved as the fallback. Operators flip the flag per-environment after backfill + cron are confirmed healthy (task_usage_rollup_lag_seconds() helper added for monitoring). 3. UpsertTaskUsage corrections invisible to rollup: added task_usage.updated_at column (default now(), backfilled from created_at), and bumped it on conflict. Corrections now mark the bucket dirty and the next window call recomputes it correctly. 4. CREATE INDEX blocking writes on hot table: split into separate single-statement migrations using CREATE INDEX CONCURRENTLY (074, 075), matching the 035/067 pattern. Also: cron.schedule() removed from migrations entirely. Migration 076 only enables the extension (gracefully on unsupported envs); the actual schedule is a documented operator runbook step that runs AFTER backfill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * fix(server): trigger-driven invalidation + online-safe migration for task_usage_daily Round-2 review feedback on PR #2256: 1. Add explicit dirty-bucket queue (task_usage_daily_dirty) populated by triggers on agent_task_queue (UPDATE OF runtime_id, DELETE) and task_usage (DELETE). The rollup window function drains both this queue and the updated_at-based discovery, so runtime reassignment and issue-cascade deletes no longer leave the rollup divergent from the raw query. Triggers join via agent (not issue) to look up workspace_id, because when the cascade comes from issue, the issue row is already gone by the time atq's BEFORE DELETE fires; agent stays alive. 2. Make migration 072 online-safe: only ADD COLUMN updated_at TIMESTAMPTZ (nullable, no default → metadata-only ALTER, no row rewrite) and a separate ALTER for SET DEFAULT now() (also metadata-only). No bulk UPDATE on the hot task_usage table. The rollup window function's dirty_keys CTE handles legacy NULL rows via an OR branch, supported by partial index idx_task_usage_created_at_legacy. 3. Refresh stale documentation in cmd/backfill_task_usage_daily/main.go header to describe the current recompute/replace semantics, idempotent re-runnability, and the actual migration numbering (072..077). Tests: - TestRollupTaskUsageDaily_InvalidationOnReassign: verifies usage moves between runtime buckets after ReassignTasksToRuntime-style update. - TestRollupTaskUsageDaily_InvalidationOnIssueDelete: verifies daily bucket is cleared after issue delete cascades through atq → task_usage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * fix(server): close dirty-queue race + move legacy partial index to its own concurrent migration Round-3 review feedback on PR #2256: 1. Blocker: dirty-queue invalidations could be silently lost under concurrency. ON CONFLICT DO NOTHING let a late trigger see the row already enqueued, no-op, and then the rollup drain (WHERE enqueued_at < p_to) would delete the original row — losing the late invalidation. Switched all three trigger enqueue paths to ON CONFLICT DO UPDATE SET enqueued_at = GREATEST(existing, EXCLUDED.enqueued_at), so any invalidation arriving during a rollup tick keeps enqueued_at > p_to (p_to = now() - 5min) and survives the post-tick drain. 2. High: idx_task_usage_created_at_legacy (partial index on hot task_usage table) was being created in the regular 077 migration without CONCURRENTLY. Moved to new migration 078 with CREATE INDEX CONCURRENTLY, matching the pattern of 074/075. 077's down migration leaves the index alone (it is owned by 078). 3. Minor: gofmt -w on runtime_rollup_test.go and backfill_task_usage_daily/main.go (tabs were lost in the original heredoc append). PR description rewritten to describe the current recompute/replace + dirty queue + feature flag design and the 072..078 migration ordering. Tests still green: TestRollupTaskUsageDaily_* (including both new invalidation regressions), TestGetRuntimeUsage_*, TestWorkspaceUsage_*. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> * fix(server): unify workspace_id source via agent in rollup window function Round-4 review feedback (J) on PR #2256: M1 (must-fix): The dirty queue triggers resolved workspace_id via `agent.workspace_id`, but the window function's `dirty_from_updates` discovery and `recomputed` recompute join used `issue.workspace_id`. There is no schema-level FK guaranteeing `agent.workspace_id == issue.workspace_id`. Any divergence (future cross-workspace task scenarios, data repairs, migration bugs) would cause: - dirty queue rows with workspace_id from agent - recompute join filtering by workspace_id from issue - 0 matches in recompute → bucket erroneously hits the deleted_empty branch and the daily row is silently dropped - dirty_from_updates path attributing usage to the wrong workspace Replaced both CTEs to JOIN agent (not issue) so trigger / discovery / recompute share one workspace_id source. Comment in 077 explains the constraint. N1: Refreshed two stale references in cmd/backfill_task_usage_daily/main.go (header now says "072..078"; stampWatermark warning now mentions migration 073, where the rollup state table is actually introduced). Test: New TestRollupTaskUsageDaily_WorkspaceMismatch constructs an atq with agent.workspace_id != issue.workspace_id, asserts the bucket lands under agent's workspace (not issue's), and re-asserts after a runtime reassign in the foreign workspace. Acts as a canary if the schema invariant changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> --------- Co-authored-by: Eve <eve@multica.ai> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: multica-agent <github@multica.ai> Co-authored-by: Devv <devv@Devvs-Mac-mini.local>	2026-05-08 15:35:21 +08:00
Bohan Jiang	d12d690c38	fix(usage): bucket workspace usage by task_usage.created_at, not enqueue time (#1176) ... GetWorkspaceUsageByDay and GetWorkspaceUsageSummary had the same date attribution bug as the runtime endpoint fixed in #1167: they bucketed and filtered on agent_task_queue.created_at (enqueue time), so a task that queued at 23:58 and reported usage at 00:05 was attributed to the prior day, and ?days=N became a rolling now()-N window that clipped the morning of the earliest day returned. Switch both queries to task_usage.created_at (~= task completion time) and snap the since cutoff to start-of-day via DATE_TRUNC, mirroring ListRuntimeUsage. These endpoints have no frontend caller today, but per offline discussion they will back the upcoming workspace-level usage dashboard. Fix preemptively so the dashboard inherits correct numbers. Add a regression test covering both endpoints with the same cross-midnight + earliest-day-cutoff scenarios used for runtime usage.	2026-04-16 19:06:49 +08:00
Bohan Jiang	88982ad23f	feat(issues): display token usage per issue in detail sidebar (#581) ... * feat(issues): display token usage per issue in detail sidebar Add a new "Token usage" section to the issue detail right sidebar that shows aggregated input/output tokens, cache tokens, and run count across all tasks for the issue. Backed by a new SQL query and API endpoint. * fix(db): add index on agent_task_queue(issue_id) for usage queries The GetIssueUsageSummary query joins agent_task_queue filtered by issue_id across all statuses. The existing partial index (migration 022) only covers queued/dispatched rows, so completed tasks require a sequential scan. Add a general index to prevent performance degradation as task volume grows.	2026-04-10 14:34:32 +08:00
Jiang Bohan	8a8d3ea20e	feat(usage): add per-task token usage tracking ... Extract token usage from Claude Code's stream-json output in real-time during task execution, replacing the inaccurate global JSONL log scanner. - New `task_usage` table: tracks (task_id, provider, model) level usage - Agent SDK: parse `message.usage` from assistant messages, accumulate per-model and return in Result - Daemon: convert agent usage to entries, send with CompleteTask - Server: store usage on task completion, expose workspace-level aggregation APIs (GET /api/usage/daily, GET /api/usage/summary)	2026-04-08 13:08:15 +08:00